Visual motion feedback for user interface

ABSTRACT

Aspects of a user interface that provides visual feedback in response to user input. For example, boundary effects are presented to provide visual cues to a user to indicate that a boundary in a movable user interface element (e.g., the end of a scrollable list) has been reached. As another example, parallax effects are presented in which multiple parallel or substantially parallel layers in a multi-layer user interface move at different rates, in response to user input. As another example, simulated inertia motion of UI elements is used to provide a more natural feel for touch input. Various combinations of features are described. For example, simulated inertia motion can be used in combination with parallax effects, boundary effects, or other types of visual feedback.

CROSS REFERENCE TO RELATED APPLICATION

This application claims the benefit of U.S. Provisional PatentApplication No. 61/304,004, filed on Feb. 12, 2010, entitled“MULTI-LAYER USER INTERFACE WITH FLEXIBLE MOVEMENT,” which isincorporated herein by reference.

BACKGROUND

The design of an effective user interface poses many challenges. Onechallenge is how to provide a user with an optimal amount of visualinformation or functionality, given the space limitations of a displayand the needs of a particular user. This challenge can be especiallyacute for devices with small displays, such as smartphones or othermobile computing devices. This is because there is often moreinformation available to a user performing a particular activity (e.g.,browsing for audio or video files in a library of files) than can fit onthe display. A user can easily become lost unless careful attention ispaid to how information is presented on the limited amount availabledisplay space. Visual cues are useful for indicating, for example, auser's location when browsing a list or other collection of data, sinceit is often not possible to show an entire collection (e.g., a list ofcontacts stored in a smartphone) on a small display.

Another challenge is how to provide a high level of functionality whilemaintaining a satisfying and consistent user experience. As devices havebecome more complex, and as consumers have become more demanding, it hasbecome increasingly difficult to design user interfaces that areconvenient and pleasing for users, without sacrificing reliability,flexibility, functionality or performance.

Whatever the benefits of previous techniques, they do not have theadvantages of the techniques and tools presented below.

SUMMARY

Techniques and tools are described that relate to different aspects of auser interface that provides visual feedback in response to user input.For example, boundary effects are presented to provide visual cues to auser to indicate that a boundary in a movable user interface element(e.g., the end of a scrollable list) has been reached. As anotherexample, parallax effects are presented in which multiple parallel orsubstantially parallel layers in a multi-layer user interface move atdifferent rates, in response to user input. As another example,simulated inertia motion of UI elements is used to provide a morenatural feel for touch input. Various combinations of features aredescribed. For example, simulated inertia motion can be used incombination with parallax effects, boundary effects, or other types ofvisual feedback.

In one aspect, a user interface (UI) system receives gesture informationcorresponding to a gesture on a touch input device. The UI systemcalculates simulated inertia motion for a movable user interface elementbased at least in part on the gesture information, and potentially onother inertia information such as a friction coefficient or a parkingspeed coefficient. Based at least in part on the gesture information andon the simulated inertia motion, the UI system calculates a post-gestureposition of the movable user interface element. The UI system determinesthat the post-gesture position exceeds a gesture boundary of the movableuser interface element, and calculates a distortion effect (e.g., asqueeze, compression or squish effect) in the movable user interfaceelement to indicate that the gesture boundary has been exceeded.Calculating the distortion effect can include, for example, determiningan extent by which the gesture boundary has been exceeded, determining acompressible area of the movable user interface element, determining ascale factor for the distortion effect based at least in part on thecompressible area and the extent by which the gesture boundary has beenexceeded, and scaling the compressible area according to the scalefactor. The distortion effect can be calculated based on a distortionpoint (which, for compression, can be referred to as a compression pointor squeeze point), which can indicate the part of the UI element to bedistorted.

In another aspect, user input (e.g., a gesture on a touch screen)indicates movement in a graphical user interface element having pluralmovable layers. Based at least in part on inertia information and theuser input, a UI system calculates a first motion having a firstmovement rate in a first layer of the plural movable layers, andcalculates a parallax motion in a second layer of the plural movablelayers. The parallax motion is based at least in part on the firstmotion (and potentially simulated inertia motion), and the parallaxmotion comprises a movement of the second layer at a second movementrate that differs from the first movement rate. The parallax motion canbe calculated based on, for example, a parallax constant for the secondlayer, or an amount of displayable data in the second layer.

In another aspect, a UI system receives gesture informationcorresponding to a gesture on a touch input device, the gestureinformation indicating a movement of a user interface element having amovement boundary. Based at least in part on the gesture information,the UI system computes a new position of the user interface element.Based at least in part on the new position, the UI system determinesthat the movement boundary has been exceeded. The UI system determinesan extent by which the movement boundary has been exceeded, determines acompressible area of the user interface element, determines a scalefactor for a distortion effect based at least in part on thecompressible area and the extent by which the movement boundary has beenexceeded, and presents a distortion effect in the user interfaceelement. The distortion effect comprises a visual compression of contentin the compressible area (e.g., text, images, graphics, video or otherdisplayable content) according to the scale factor. Depending, forexample, on the size of the compressible area and the size of thedisplay area, some parts of the compressible area may not be visible ona display, so the distortion can be virtual (e.g., in areas that are notvisible on a display) or the distortion can be actually displayed, orsome parts of the distorted content can be displayed while other partsof the distorted content are not displayed. The visual compression is ina dimension that corresponds to the movement of the user interfaceelement. For example, a vertical movement in a UI element that exceeds amovement boundary can cause content in the UI element to be verticallycompressed or squeezed.

The foregoing and other objects, features, and advantages of theinvention will become more apparent from the following detaileddescription, which proceeds with reference to the accompanying figures.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A-1C and 2 are flow charts showing example techniques forpresenting motion feedback in user interface elements, according to oneor more described embodiments.

FIG. 3 is a diagram showing a boundary effect, according to one or snoredescribed embodiments.

FIGS. 4A-4C are diagrams showing parallax effects, according to one ormore described embodiments.

FIGS. 5 and 6A-6E are diagrams showing parallax effects and boundaryeffects in a user interface having parallel layers, according to one ormore described embodiments.

FIGS. 7A, 7B, 8A and 8B are diagrams showing gesture boundary areaswhich can be used to determine whether to present boundary effects,according to one or more described embodiments.

FIG. 9 is a diagram showing example pinch and stretch gestures,according to one or more described embodiments.

FIG. 10 is a graph showing changes in position over time of a UI elementthat exhibits a boundary feedback effect, according to one or moredescribed embodiments.

FIG. 11 is a system diagram showing a UI system in which describedembodiments can be implemented.

FIG. 12 illustrates a generalized example of a suitable computingenvironment in which several of the described embodiments may beimplemented.

FIG. 13 illustrates a generalized example of a suitable implementationenvironment in which one or more described embodiments may beimplemented.

FIG. 14 illustrates a generalized example of a mobile computing devicein which one or more described embodiments may be implemented.

DETAILED DESCRIPTION

Techniques and tools are described that relate to different aspects of auser interface that provides visual feedback in response to user input.For example, boundary effects are presented to provide visual cues to auser to indicate that a boundary in a movable user interface element(e.g., the end of a scrollable list) has been reached. As anotherexample, parallax effects are presented in which multiple parallel orsubstantially parallel layers in a multi-layer user interface move atdifferent rates, in response to user input. As another example,simulated inertia motion of UI elements is used to provide a morenatural feel for touch input. Various combinations of features aredescribed. In one implementation, a UI system that accepts touch inputincludes detailed motion rules (e.g., rules for interpreting differentkinds of touch input, rules for presenting inertia motion in UI elementsin response to touch input, rules for determining boundaries in UIelements, etc.). The motion rules can be combined with variouscombinations of optional motion features such as parallax effects,boundary effects, and other visual feedback. The visual feedback that ispresented according to motion rules and optional motion features in a UIelement can depend on many factors, such as the type of the UI elementand the content of the UI element.

Various alternatives to the implementations described herein arepossible. For example, techniques described with reference to flowchartdiagrams can be altered by changing the ordering of stages shown in theflowcharts, by repeating or omitting certain stages, etc. As anotherexample, systems described with reference to system diagrams can bealtered by changing the ordering of processing stages shown in thediagrams, by repeating or omitting certain stages, etc. As anotherexample, user interfaces described with reference to diagrams can bealtered by changing the content or arrangement of user interfacefeatures shown in the diagrams, by omitting certain features, etc. Asanother example, although some implementations are described withreference to specific devices and user input mechanisms (e.g., mobiledevices with a touchscreen interface), described techniques and toolscan be used with other devices and/or user input mechanisms.

The various techniques and tools can be used in combination orindependently. Different embodiments implement one or more of thedescribed techniques and tools.

I. Overview of Motion Feedback Features for User Interfaces

As devices have become more complex, and as consumers have become snoredemanding, it has become increasingly difficult to design userinterfaces that are convenient and pleasing for users, withoutsacrificing reliability, flexibility, functionality or performance. Thefeel of a user interface (UI) is becoming increasingly important todistinguish the underlying product from its competitors. An importantcontributor to the feel of a UI is how it reacts when a user interactswith it. This is especially true for touch-based interfaces.

Accordingly, techniques and tools are described for providing feedback(e.g., visual cues such as parallax effects, boundary effects, etc.) tousers in response to user input (e.g., touch input). In someembodiments, movements in elements (also referred to as “controls”) arebased at least in part on user input (e.g., gestures on a touchscreen)and an inertia model. For example, a movement in a UI element can beextended beyond the actual size of a gesture on a touchscreen byapplying inertia to the movement. Applying inertia to a movement in a UIelement typically involves performing one more calculations usinggesture information (e.g., a gesture start position, a gesture endposition, gesture velocity and/or other information) and one or moreinertia motion values (e.g., friction coefficients) to determine apost-gesture state (e.g., a new position) for the UI element. Simulatedinertia motion can be used in combination with other effects (e.g.,parallax effects, boundary effects, etc.) to provide feedback to a user.In any of the examples herein, movements in UI elements can be renderedfor display (e.g., depicting calculated distortion, parallax, or othereffects, if any).

Movement in UI elements typically depends to some extent on userinteraction. For example, a user that wishes to navigate from one partof a UI element to another (e.g., from the beginning of a scrollablelist to the end of the list) provides user input to indicate a desiredmovement. The user input can then cause movement in the UI element andpotentially other elements in the user interface. In some embodiments, auser causes movement in a display area of a device by interacting with atouchscreen. The interaction can include, for example, contacting thetouchscreen with a fingertip, stylus or other object and moving it(e.g., with a flicking or sweeping motion) across the surface of thetouchscreen to cause movement in a desired direction. Alternatively, auser can interact with a user interface in some other way, such as bypressing buttons (e.g., directional buttons) on a keypad or keyboard,moving a trackball, pointing and clicking with a mouse, making a voicecommand, etc.

The actual amount and direction of the user's motion that is necessaryto produce particular movements can vary depending on implementation oruser preferences. For example, a user interface system can include adefault setting that is used to calculate the amount of motion (e.g., interms of pixels) as a function of the size or rate of a user movement.As another example, a user can adjust a touchscreen sensitivity control,such that the same motion of a fingertip or stylus on a touchscreen willproduce smaller or larger movements, depending on the setting of thecontrol. Gestures can be made in various directions to cause movement inUI elements. For example, upward and downward gestures can cause upwardor downward movements, respectively, while rightward and leftwardmovements can cause rightward and leftward movements, respectively.Upward/downward motion can even be combined with left/right motion fordiagonal movements. Other kinds of motion, such as non-linear motion(e.g., curves) or bi-directional motion (e.g., pinch or stretch motionsmade with multiple contact points on a touchscreen) also can be used tocause movement in UI elements.

Example 1 Inertia, Boundary Effects and Parallax Effects: Overview

FIG. 1A is a flow chart showing a general technique 100 for providingmotion feedback in a UI. At 101, a device receives user input indicatingmotion in a UI element. For example, a UI system on a mobile devicereceives gesture information corresponding to a gesture on a touchscreenon the mobile device. At 102, the device determines whether inertia willbe applied to the motion indicated by the user input. For example, a UIsystem determines based on gesture information (e.g., gesture startposition, gesture end position, gesture direction, gesture velocity)whether to apply inertia to the motion in the UI element. At 103, thedevice determines whether visual effects (e.g., boundary effects,parallax effects, etc.) will be applied to the motion indicated by theuser input. For example, the device determines whether to apply adistortion effect (e.g., a compression or squeeze effect) to indicatethat a boundary in the UI element (e.g., a boundary at the end of ascrollable list) has been reached. As another example, the devicedetermines whether to apply a parallax effect (e.g., by moving parallellayers in a multi-layer UI element at different rates). The appliedeffects also can be based on inertia, where inertia is applied to themotion indicated by the user input. For example, if a UI system appliesinertia to a movement and calculates, based on the inertia, a newposition for a UI element that is outside a boundary for the UI element,the UI system can apply a boundary effect to provide a visual indicatorthat the boundary has been reached. At 104, the motion in the UI elementis rendered for display.

FIG. 19 is a flow chart showing a technique 110 for providing boundaryeffects in combination with inertia motion. At 111, a UI system receivesgesture information corresponding to a gesture. For example, the UIsystem receives gesture coordinates and velocity information for thegesture. At 112, the UI system calculates inertia motion based on thegesture information. For example, the UI system determines that inertiamotion is applied based on the velocity information, and calculates aduration of inertia motion for the gesture. At 113, the UI systemcalculates a post-gesture position based on the gesture information andthe inertia motion. For example, the UI system calculates thepost-gesture position based on the gesture coordinates and the durationof the inertia motion. At 114, the UI system determines that a boundaryfor the UI element has been exceeded. For example, the UI systemcompares one or more coordinates (e.g., vertical or horizontalcoordinates) of the post-gesture position and determines an extent bywhich the post-gesture position exceeds the boundary. At 115, the UIsystem calculates a distortion effect to indicate that the boundary hasbeen exceeded. For example, the UI system calculates a squeeze orcompression effect in the content of the UI element based on the extentto which the post-gesture position exceeds the boundary.

FIG. 1C is a flow chart showing a technique 120 for providing parallaxeffects in combination with inertia motion. At 121, a UI system receivesuser input indicating motion in a UI element having plural layers. Forexample, the UI system receives gesture coordinates and velocityinformation for a gesture on a touch screen, where the gesture isdirected to a content layer in multi-layer UI. At 122, the UI systemcalculates motion in a first layer based on inertia information and theuser input. For example, the UI system determines that inertia motionshould be applied to movement in the content layer based on the velocityinformation, and calculates a duration of inertia motion for themovement. At 123, the UI system calculates a parallax motion in a secondlayer based on the first motion in the first layer. For example, the UIsystem calculates the parallax motion in a layer above the content layerbased on the motion in the content layer, with the parallax motionhaving a different movement rate than the motion in the content layer.The parallax motion also can include inertia motion, or inertia motioncan be omitted in the parallax motion.

In any of the above techniques, any combination of the inertia,boundary, parallax, distortion, and other effects described herein canbe applied. Depending on implementation and the type of processingdesired, processing stages shown in example techniques 100, 110, 120 canbe rearranged, added, omitted, split into multiple stages, combined withother stages, and/or replaced with like stages.

Example 2 Inertia, Boundary Effects and Parallax Effects: DetailedTechnique

FIG. 2 is a flow chart showing a detailed example technique 200 forproviding visual feedback in a UI in response to a user gesture.

At 210, a UI system on a device receives touch input information in atouch input stream. For example, the touch input stream comprises datacorresponding to a gesture on a touchscreen of a mobile device. Datareceived from the touch input stream can include, for example, gestureinformation such as a gesture start position, a gesture end position,and timestamps for the gesture. The touch input stream is typicallyreceived from a device operating system, which converts raw datareceived from a touch input device (e.g., a touchscreen) into gestureinformation. Alternatively, data received from the touch input streamcan include other information, or gesture information can be receivedfrom some other source.

At 212, filtering is applied to the touch input stream. In the filteringstage, one or more algorithms are applied to the touch input streamcoming from the OS to filter out or correct anomalous data. For example,the filtering stage can correct misaligned touch data caused byjittering (e.g., values that are not aligned with previous inputs) orfilter out spurious touch contact points (e.g., incorrect interpretationof a single touch point as multiple touch points that are closetogether), etc. As another example, if only single-touch-point gesturesare allowed, the filtering stage can convert any multi-touch input intoa single-touch input. Alternatively, touch input filtering can beperformed during generating of the touch input stream (e.g., at thedevice OS). As another alternative, touch input filtering can beperformed during a coordinate space transform stage (e.g., coordinatespace transform 220). As another alternative, touch input filtering canbe omitted.

At 220, the UI system applies a coordinate space transform to data inthe touch input stream corresponding to the gesture. For example, acoordinate space transform is applied to the data from the touch inputstream in order to account for possible rotations of the device, scalechanges, influence from other animations, etc., in order to properlyinterpret the original input stream. For example, if a UI element isrotated 90 degrees such that vertical movement in the UI element becomeshorizontal movement (or vice versa), a vertical gesture can betransformed to a horizontal gesture (or vice versa) to account for therotation of the device. If no adjustments are necessary, the coordinatespace transform can leave gesture information unchanged. Alternativelythe coordinate space transform state can be omitted.

At 230, the UI system calculates the velocity at the end of the gesture.For example, the velocity is calculated by determining a first positionnear the end of the gesture and an end position of the gesture, anddividing by the time elapsed during the movement from the first positionnear the end of the gesture to the end position. In one implementation,the first position is determined by finding the gesture position atapproximately 100 ms prior to the end of the gesture. Measuring velocitynear the end of the gesture can help to provide a more accurate motionresulting from the gesture than measuring velocity over the entirecourse of the gesture. For example, if a gesture starts slowly and endswith a higher velocity, measuring the velocity at the end of the gesturecan help to more accurately reflect the user's intended gesture (e.g., astrong flick). Alternatively, the velocity is calculated by determiningthe distance (e.g., in pixel units) between the start position for thegesture and the end position of the gesture, and dividing by the timeelapsed during the movement from the start position to the end position.The time elapsed can be calculated, for example, by computing thedifference between a timestamp associated with the start position and atimestamp associated with the end position.

At 240, the UI system determines whether the gesture is an inertiagesture. As used herein, an inertia gesture refers to a gesture, such asa flick gesture, capable of causing movement in one or more userinterface elements to which inertia can be applied. The UI system candistinguish between a non-inertia gesture and an inertia gesture bydetermining how quickly the user's finger, stylus, etc., was moving whenit broke contact with the touchscreen, and whether the velocity exceedsa threshold. If the gesture ends with a velocity above the threshold,the gesture can be interpreted as an inertia gesture. For example, agesture that starts with panning motion at a velocity below thethreshold and ends with a velocity above the threshold can beinterpreted as ending with a flick that causes movement to which inertiacan be applied. If the gesture ends with a velocity below the threshold,the gesture can be interpreted as a non-inertia gesture. Exemplarytechniques and tools used in some implementations for gestureinterpretation are described in detail below.

If the gesture is an inertia gesture (e.g., a flick gesture), at 250 theUI system determines whether inertia will be applied to the motionindicated by the gesture. For example, the UI system determines based ongesture information (e.g., end-of-gesture velocity) and/or otherinformation (e.g., user preferences) whether to apply inertia to themotion in the UI element. Despite being considered an inertia gesture, agesture such as a flick may still not have inertia applied to itsresulting movements, such as when a flick gesture is received for a UIelement that does not support inertia movements, or for a UI element forwhich inertia movement has been deactivated (e.g., according to userpreference).

If inertia is not to be applied (e.g., when the gesture is not aninertia gesture), at 254 the UI system computes a new position for theUI element based on gesture information (e.g., end-of-gesture positioncoordinates). If inertia is to be applied, at 252 the UI system computesa new position based on the gesture information (e.g., end-of-gestureposition coordinates) and simulated inertia. For example, the simulatedinertia involves treating a UI element, or part of a UI element, as aphysical object of non-zero mass that moves according to anapproximation of Newtonian physics. The approximation can include, forexample, a friction coefficient and/or other parameters that control howthe movement is calculated and/or rendered.

When the new position of the UI element has been computed (with orwithout simulated inertia), the UI system determines at 260 whetherboundary feedback will be presented. Determining whether boundaryfeedback will be presented involves determining whether the new positionis within boundaries (if any) of the UI element. For example, in ascrollable list, the UI system can determine whether the new position iscalculated to be outside the boundaries of the scrollable list (e.g.,below the end of a vertically scrollable list). Some UI elements may nothave boundaries that can be exceeded by any permitted motion. Forexample, a UI element may take the form of a wrappable list, which mayhave a default entry position but no beginning or end. If the wrappablelist is axis-locked (e.g., if movement is only permitted along avertical axis for a vertically scrolling list), the list may have noboundaries that can be exceeded by any permitted motion. For UI elementswithout any boundaries, or without boundaries that can be exceeded bypermitted motion, the determination of whether the new position iswithin boundaries can be skipped. Axis locking is described in moredetail below.

If boundary feedback is to be presented, at 262 the UI system applies aboundary effect to the UI element. For example, the UI system can applya visual distortion effect such as a “squish” or compression of text,images or other visual information in the UI element, to provide avisual cue that a boundary of the UI element has been reached. Boundaryeffects are described in more detail below.

The UI system determines at 270 whether parallax feedback will bepresented. Determining whether parallax feedback will be presentedinvolves determining whether the UI element has multiple parallel layersor substantially parallel layers that can be moved at different ratesbased on the same gesture. If parallax feedback is to be presented, at272 the UI system applies a parallax effect to the UI element. Ingeneral, a parallax effect involves movement of multiple parallellayers, or substantially parallel layers, at different rates. Exampleparallax effects are described in more detail below.

The processing stages in example technique 200 indicate example flows ofinformation in a UI system. Depending on implementation and the type ofprocessing desired, processing stages can be rearranged, added, omitted,split into multiple stages, combined with other stages, and/or replacedwith like stages.

For example, although example technique 200 shows stages of receivingdata from a touch input stream, applying touch input filtering, applyinga coordinate space transform, calculating a velocity at the end of agesture, and determining whether the gesture is an inertia gesture, suchprocessing stages are only exemplary. Gesture information (e.g., gesturevelocity, position, whether the gesture is a candidate for simulatedinertia, etc.) can be obtained in other ways. As an example, a modulethat determines whether to apply inertia motion and determines whetherto apply boundary feedback or parallax effects can obtain gesture datafrom another source, such as another module that accepts touch input andmakes calculations to obtain gesture information (e.g., gesturevelocity, end-of-gesture position).

As another example, although example technique 200 shows a determinationof whether to present boundary feedback occurring before a determinationof whether to present parallax feedback, such an arrangement is onlyexemplary. A determination of whether to present boundary feedbackand/or parallax feedback can be performed in other ways. As examples,once a new position has been calculated, determinations of whether topresent boundary feedback and/or parallax feedback can occur inparallel, or the determination of whether to present a parallax effectcan occur before the determination of whether to present a boundaryeffect. Such arrangements can be useful, for example, where a gesturemay cause movements in multiple parallel layers of a UI element prior toreaching a boundary of the element. A UI system also can determine(e.g., based on characteristics of a current UI element) whetherboundary effects and/or parallax effects are not available (e.g., for UIelements that do not have multiple layers or boundaries), and skipprocessing stages that are not relevant.

II. Boundary Effects

Boundary feedback can be used to provide visual cues to a user toindicate that a boundary (e.g., a boundary at the end, beginning, orother location) in a UI element (e.g., a data collection such as a list)has been reached. In described implementations, a UI system presents aboundary effect in a UI element (or a portion of a UI element) bycausing the UI element to be displayed in a visually distorted state,such as a squeezed or compressed state (i.e., a state in which text,images or other content is shown to be smaller than normal in one ormore dimensions), to indicate that a boundary of the UI element has beenreached.

Described techniques and tools for presenting boundary feedback can beapplied to any UI element with one or more boundaries that can bemanipulated by moving the element. For example, described techniques andtools can be used in an email viewer, such that text in a scrollableemail message is distorted (e.g., squeezed or compressed) to indicatethat the end of the email message has been reached.

Boundary effects (e.g., distortion effects) can be presented indifferent ways. For example, a boundary effect can be held in place fordifferent lengths of time depending on user input and/or design choice.A boundary effect can end, for example, by returning the UI element to anormal (e.g., undistorted) state when a user lifts a finger, stylus orother object to end an interaction with a touchscreen after reaching aboundary, or when an inertia motion has completed. As another example,distortion effects other than a squish, squeeze or compression can beused. One alternative distortion effect is a visual stretch. A stretcheffect can be used, for example, in combination with a snap-backanimation to indicate that boundary has been reached.

Boundary effects can be presented even when it is possible to continue amovement beyond a boundary. For example, if a user scrolls to the end ofa vertically-oriented list, causing a distortion of text or images atthe end of the list, further motion can cause the list to wrap past theboundary and back to the beginning of the list. The UI also can show anelement (or part of an element) at the top of the list to indicate thatfurther movement can allow the user to wrap back to the beginning of thelist.

Example 3 Boundary Effect: Distortion

FIG. 3 is a diagram showing a graphical user interface (GUI) presentedby a UI system that uses a distortion effect to indicate that a boundaryof UI element has been reached. According to the example shown in FIG.3, a user 302 (represented by the hand icon) interacts with a listcomprising list elements (“Contact1,” “Contact2,” etc.). In thisexample, distortion effects depend at least in part on the location of asqueeze point 396. Some list elements with distortion effects are shownas being outside display area 300.

FIG. 3 shows example states 390-394. In state 390, user 302 interactswith a touchscreen by making an upward motion, indicated by an initialgesture position 350 and an end-of-gesture touch position 352. Theinteraction can include, for example, contacting the touchscreen with afingertip, stylus or other object and moving it (e.g., with a flickingor sweeping motion) along the surface of the touchscreen. Although FIG.3 shows user 302 interacting with the touchscreen at particularlocations in the display area 300, the UI system allows interaction withother parts of the touchscreen to cause movement in the list.Furthermore, although the example shown in FIG. 3 shows user 302 makingan upward motion to scroll towards the end of the list, user 302 alsocan make other motions (e.g., downward motions to scroll towards thebeginning of the list). The UI system can interpret different kinds ofupward or downward user movements, even diagonal movements extending tothe right or left of the vertical plane, as a valid upward or downwardmotion.

From state 390, the upward motion causes a distortion effect shown instate 392. In this example, the upward motion is finger-tracking motioncaused by a drag gesture, but distortion effects also can be caused byother motion resulting from other kinds of gestures, such as inertiamotion caused by a flick gesture. The distortion effect indicates that aboundary in the list has been reached. In the example shown in stateFIG. 3, the entire list is treated as a single surface, as indicated bythe single dimension line to the right of the list in states 390, 392and 394, respectively. In state 392, the list has been squeezed orcompressed in a vertical dimension, as shown by the reduced length ofthe dimension to the right of the list. The text of each list elementhas been squeezed or compressed in a vertical dimension. The elementsare distorted proportionally. The effect in state 392 is as if all thelist elements are being compressed against a barrier at the squeezepoint 396.

In the example shown in state 392, the squeeze point 396 is indicated atthe top of a list, outside the display area 300. Other squeeze pointsare also possible. For example, the squeeze point could be at the centerof a list (e.g., at item 50 in a 100 item list) or at the top of avisible portion of a list. In this example, the list can be consideredas having two parts—one part above the squeeze point, and one part belowthe squeeze point—where only one part of the list is squeezed. Thesqueeze point can change dynamically, depending on the state of the listand/or display. For example, a squeeze point can move up or down (e.g.,in response to where the center of the list is) as elements are added toor removed from the list, or a squeeze point can update automatically(e.g., when the end of the list has been reached) to be at the top of avisible portion of the list. As another example, a squeeze point can beplaced outside of a list. This can be useful to provide more consistentvisual feedback, such as when a UI element does not fill the visiblearea.

In state 394, the list has returned to the undistorted state shown instate 390. For example, the list can return to the undistorted stateafter the gesture shown in state 390 is ended (e.g., when the userbreaks contact with the touchscreen).

The upward motion shown in FIG. 3 is only an example of a possible userinteraction. The same motion and/or other user interactions (e.g.,motions having different sizes, directions, or velocities) can causedifferent effects, different display states, different transitionsbetween display states, etc. For example, a motion that causes adistortion effect in a UI element (e.g., at the end of a verticallyscrollable list) also can cause another portion of the UI element (e.g.,a list item at the beginning of a vertically scrollable list) to bedisplayed to indicate availability of a wrapping feature in the list.Further movement can then cause wrapping in the UI element (e.g., fromthe end back to the beginning of a vertically scrollable list).

States 390-394 are only examples of possible states. In practice, a UIelement can exist in any number of states (e.g., in intermediate statesbetween example states 390-394, etc.) in addition to, or as alternativesto, the example states 390-394. For example, it is preferable to show agradual transition between an undistorted state (e.g., state 390) and adistorted state (e.g., state 392), or from a distorted state to anundistorted state, to provide a more natural feel and avoid theappearance of abrupt changes in the display. Intermediate states, suchas states that may occur between state 390 and state 392, or betweenstate 392 and state 394 can show gradually increasing or decreasingdegrees of distortion, as appropriate.

III. Parallax Effects

In described embodiments, a UI system can present parallel, orsubstantially parallel, movable layers. The UI system can present aparallax effect, in which layers move at different speeds relative toone another. The effect is referred to as a parallax effect because, ina typical example, a layer that is of interest to a user moves at afaster rate than other layers, as though the layer of interest werecloser to the user than the other, slower-moving layers. However, theterm “parallax effect” as used herein refers more generally to effectsin which layers move at different rates relative to one another.

The rate of movement in each layer can depend on several factors,including the amount of data to be presented visually (e.g., text orgraphics) in the layers, or the arrangement of the layers relative toone another. The amount of data to be presented visually in a layer canmeasured by, for example, determining the length as measured in ahorizontal direction of the data as rendered on a display or as laid outfor possible rendering on the display. Length can be measured in pixelsor by some other suitable measure (e.g., the number of characters in astring of text). A layer with a larger amount of data and moving at afaster rate can advance by a number of pixels that is greater than alayer with a smaller amount of data moving at a slower rate. Layermovement rates can be determined in different ways. For example,movement rates in slower layers can be derived from movement rates infaster layers, or vice versa. Or, layer movement rates can be determinedindependently of one another. Layers that exhibit parallax effects canbe overlapping layers or non-overlapping layers.

When user interaction causes movement in layers, the movement of thelayers is a typically a function of the length of the layers and thesize and direction of the motion made by the user. For example, aleftward flicking motion on a touchscreen produces a leftward movementof the layers relative to the display area. Depending on implementationand/or user preferences, user input can be interpreted in different waysto produce different kinds of movement in the layers. For example, a UIsystem can interpret any movement to the left or right, even diagonalmovements extending well above or below the horizontal plane, as a validleftward or rightward motion of a layer, or the system can require moreprecise movements. As another example, a UI system can require that auser interact with a part of a touchscreen corresponding to the displayarea occupied by a layer before moving that layer, or the system canallow interaction with other parts of the touchscreen to cause movementin a layer. As another example, a user can use an upward or downwardmotion to scroll up or down in a part of the content layer that does notappear on the display all at once.

In some embodiments, lock points indicate corresponding positions inlayers with which a display area of a device will be aligned. Forexample, when a user navigates to a position on a content layer suchthat the left edge of the display area is at a left-edge lock point “A,”the left edge of display area will also be aligned at a correspondingleft-edge lock point “A” in each of the other layers. Lock points alsocan indicate alignment of a right edge of a display area (right-edgelock points), or other types of alignment (e.g., center lock points).Typically, corresponding lock points in each layer are positioned toaccount for the fact that layers will move at different speeds. Forexample, if the distance between a first lock point and a second lockpoint in a content layer is twice as great as the distance betweencorresponding first and second lock points in a background layer, thebackground layer moves at half the rate of the content layer whentransitioning between the two lock points.

In addition to indicating corresponding positions in layers, lock pointscan exhibit other behavior. For example, lock points can indicatepositions in a content layer to which the layer will move when the partof the layer corresponding to the lock point comes into view on thedisplay. This can be useful, for example, when an image, list or othercontent element comes partially into view near an edge of the displayarea—the content layer can automatically bring the content elementcompletely into view by moving the layer such that an edge of thedisplay area aligns with an appropriate lock point. A lock animation canbe performed at the end of a gesture, such as a flick or pan gesture, toalign the layers with a particular lock point. As an example, a lockanimation can be performed at the end of a gesture that causes movementof a content layer to a position between two elements in a content layer(e.g., where portions of two images in a content layer are visible in adisplay area). A UI system can select an element (e.g., by checkingwhich element occupies more space in the display area) and transition tofocus on that element using the lock animations. This can improve theoverall look of the layers and can be effective in bringing informationor functional elements into view in a display area. A lock animationalso can be used together with simulated inertia motion. For example, alock animation can be presented after inertia motion stops, or a lockanimation can be blended with inertia motion (such as by extendinginertia motion to a lock point, or ending inertia motion early bygradually coming to a stop at a lock point) to present a smoothtransition to a lock point.

The amounts and rates of movements presented in parallax effects can becalculated and presented in different ways. In a detailed exampledescribed below, equations are described for calculating parallax effectmovements in which a parallax constant is used to determine anewposition for a layer after a gesture. As another example, motion inlayers and/or other elements, such as lists, can be calculated based onmotion ratios. For example, a UI system can calculate motion ratios fora background layer and a title layer by dividing the width of thebackground layer and the width of the title layer, respectively, by amaximum width of the content layer. Taking into account the widths ofthe background layer and the title layer, a system can map locations oflock points in the background layer and the title layer, respectively,based on the locations of corresponding lock points in the contentlayer.

Movement of various layers can differ depending on context. For example,a user can navigate left from the beginning of a content layer to reachthe end of a content layer, and can navigate right from the end of thecontent layer to reach the beginning of a content layer. This wrappingfeature provides more flexibility when navigating through the contentlayer. Wrapping can be handled by the UI system in different ways. Forexample, wrapping can be handled by producing an animation that shows arapid transition from the end of layers such as title layers orbackground layers back to the beginning of such layers, or vice-versa.Such animations can be combined with ordinary panning movements in thecontent layer, or with other animations in the content layer, such aslocking animations. However, wrapping functionality is not required.

Example 4 Parallax Effects: Multiple Layers with Background Layer

FIGS. 4A-4C are diagrams showing a GUI presented by a UI system withthree layers 410, 412, 414 and a background layer 450. In this example,a user 302 (represented by the hand icon) interacts with content layer414 by interacting with a touchscreen having a display area 300.

Background layer 450 floats behind the other layers. Data to bepresented visually in background layer 450 can include, for example, animage that extends beyond the boundaries of display area 300. Thecontent layer 414 includes content elements (e.g., images) 430A-H.Layers 410, 412 include text information (“Category” and “SelectedSubcategory,” respectively). The length of content layer 414 isindicated to be approximately twice the length of layer 412, which is inturn indicated to be approximately twice the length of layer 410. Thelength of background layer 450 is indicated to be slightly less than thelength of layer 412.

In FIGS. 4A-4C, the direction of motion that can be caused in the layers410, 412, 414, 450 by user 302 is indicated by a left-pointing arrow anda right-pointing arrow. These arrows indicate possible movements (leftor right horizontal movements) of layers 410, 412, 414, 450 in responseto user movements. In this example, the system interprets user movementsto the left or right, even diagonal movements extending above or belowthe horizontal plane, as a valid leftward or rightward motion of alayer. Although FIGS. 4A-4C show user 302 interacting with a portion ofdisplay area 300 that corresponds to content layer 414, the system alsoallows interaction with other parts of the touchscreen (e.g., parts thatcorrespond to portions of display area 300 occupied by other layers) tocause movement in layers 410, 412, 414, 450.

When user input indicates a motion to the right or left, the systemproduces a rightward or leftward movement of the layers 410, 412, 414,450 relative to display area 300. The amount of movement of layers 410,412, 414, 450 is a function of the data in the layers and the size orrate of the motion made by the user.

In FIGS. 4A-4C, example left-edge lock points “A,” “B” and “C” areindicated for layers 410, 412. 414. 450. The left-edge lock pointsindicate the corresponding position of the left edge of the display area300 on each layer. For example, when a user navigates to a position oncontent layer 414 such that the left edge of display area 300 is at lockpoint “A,” the left edge of display area 300 will also be aligned atlock point “A” of the other layers 410, 412, 450, as shown in FIG. 4A.In FIG. 4B, the left edge of display area 300 is at lock point “B” ineach of the layers 410, 412, 414, 450. In FIG. 4C, the left edge of thedisplay area 300 is at lock point “C” in each of the layers 410, 412,414, 450.

The lock points shown in FIGS. 4A-4C are not generally representative ofa complete set of lock points, and are limited to lock points “A,” “B”and “C” only for brevity. For example, left-edge lock points can be setfor each of the content elements 430A-430H. Alternatively, fewer lockpoints can be used, or lock points can be omitted. As anotheralternative, lock points can indicate other kinds of alignment. Forexample, right-edge lock points can indicate alignment with the rightedge of display area 300, or center lock points can indicate alignmentwith the center of display area 300.

In this example, layers 410, 412, 414, 450 move according to thefollowing rules, except during wrapping animations:

-   -   1. Content layer 414 will move at approximately twice the rate        of layer 412, which is approximately half the length of layer        414.    -   2. Layer 412 will move at approximately twice the rate of layer        410, which is approximately half the length of layer 412.    -   3. Content layer 414 will move at approximately four times the        rate of layer 410, which is approximately ¼ the length of layer        414.    -   4. Background layer 450 will move slower than layer 410.        Although background layer 450 is longer than layer 410, the        distance to be moved between neighboring lock points (e.g., lock        points “A” and “B”) in layer 410 is greater than the distance        between the corresponding lock points in background layer 450.

Movement of layers 410, 412, 414, 450 may differ from the rulesdescribed above in some circumstances. In this example, wrapping ispermitted. User 302 can navigate left from the beginning of contentlayer 414 (the position shown in FIG. 4A), and can navigate right fromthe end of content layer 414 (the position shown in FIG. 4C). During awrapping animation, some layers may move faster or slower than duringother kinds of movements. In this example, the image in background layer450 and the text in layers 410 and 412 moves faster when user inputcauses wrapping back to the beginning of content layer 414. In FIG. 4C,display area 300 shows portions of one and two letters, respectively, inlayers 410 and 412, at the end of the respective text strings. Displayarea 300 also shows the rightmost portion of the image in backgroundlayer 450. A wrapping animation to return to the state shown in FIG. 4Acan include bringing the leftmost portion of the image in backgroundlayer 450 and the beginning of the text in layers 410, 412 into viewfrom the right. This results in a more rapid movement in layers 410, 412and 450 than in other contexts, such as the transition from the stateshown FIG. 4A to the state shown in FIG. 4B.

Example 5 Inertia Motion with Parallax Effects and Boundary Effects

Parallax effects can be used in combination with boundary effects andinertia motion. For example, boundary effects can be used to indicatewhen a user has reached a boundary of a layer, or a boundary of anelement within a layer. As another example, inertia motion can be usedto extend motion of UI elements caused by some gestures (e.g., flickgestures). If inertia motion causes movement of a UI element (e.g., alayer) to extend beyond a boundary, a UI system can present a boundaryeffect.

FIG. 5 is a diagram showing two layers 530, 532. Display area 300 isindicated by a dashed line and has dimensions typical of displays onsmartphones or similar mobile computing devices. The content layer 532includes content elements 540-544. In this example, each content element540-544 comprises an image representing a music album, and textindicating the title of the respective album. The list header layer 530includes a text string (“Albums”). According to the example shown inFIG. 5, a user 302 (represented by the hand icon) interacts with contentlayer 532 by interacting with a touchscreen having the display area 300.The interaction can include, for example, contacting the touchscreenwith a fingertip, stylus or other object and moving it (e.g., with aflicking or sweeping motion) across the surface of the touchscreen.

FIG. 5 shows example display states 590-594. In display state 590, user302 interacts with a touchscreen by making a flick gesture 510, which isindicated by a leftward-pointing arrow. The flick gesture 510 causes aninertia motion in content layer 532, which continues to move after thegesture 510 has ended. Although FIG. 5 shows user 302 interacting withthe touchscreen at a particular location in the display area 300, the UIsystem allows interaction with other parts of the touchscreen to causemovement. Furthermore, although the example shown in FIG. 5 shows user302 making a leftward flick gesture, user 302 also can make othermotions (e.g., rightward motions to scroll towards the beginning of thelist). The UI system can interpret different kinds of leftward orrightward user movements, even diagonal movements extending below orabove the horizontal plane, as a valid leftward or rightward motion.

In response to the flick gesture 510, the UI system produces leftwardmovement of the layers 530, 532 relative to the display area 300. Forexample, from display state 590, the flick gesture 510 causes a leftwardmovement in the layers and leads to display state 592, in which element540 is no longer visible, and elements 542 and 544 have moved to theleft. The text string (“Albums”) in the list header layer 530 also hasmoved to the left, but at a slower rate (in terms of pixels) than thecontent layer 532. The movement of the layers 530, 532 is a function ofthe data in the layers and the velocity of the flick gesture 510.

From display state 592, the inertia motion causes continued leftwardmovement of the layers 530, 532 without further input from the user 302,and leads to display state 594 in which element 542 is no longervisible. The inertia motion causes the content layer to extend beyond aboundary (not shown) to the right of the element 544 in the contentlayer 532, which results in a distortion effect in which an image andtext in element 544 is squeezed or compressed in a horizontal dimension.The compression is indicated by the reduced length of the dimensionlines above the image and text (“Rock & Roll Part in”) of element 544,respectively. The text string (“Albums”) in the list header layer 530also has moved to the left, but at a slower rate (in terms of pixels)than the content layer 532. The text in list header layer 530 isuncompressed. The distortion effect gives user 302 an indication thatthe end of the content layer 532 has been reached.

Although a motion that is calculated to extend beyond a boundary mayresult in a distortion effect, the boundary need not prevent furthermovement in the direction of the motion. For example, if wrappingfunctionality is available, further movement beyond the boundary cancause the content layer 530 to wrap back to the beginning (e.g., back todisplay state 590). In state 594, element 540 at the beginning of thecollection is partially visible, indicating that wrapping is available.

The display can return from display state 594 to display state 592,transitioning from a display state with a distortion effect to anundistorted display state. This can occur, for example, without anyadditional input by the user. The length of time that it takes totransition between states can vary depending on implementation.

Flick gesture 510 is only an example of a possible user interaction. Thesame gesture 510 and/or other user interactions (e.g., motions havingdifferent sizes, directions, or velocities) can cause different effects,different display states, different transitions between display states,etc. Some display states (e.g., display state 594) may occur only if agesture results in a post-gesture position that is calculated to gobeyond a boundary for the layer.

Display states 590-594 are only examples of possible display states. Inpractice, a display can exist in any number of states (e.g., inintermediate states between example states 590-594, in states withdifferent visible UI elements, etc.) in addition to, or as alternativesto, the example display states 590-594. For example, it is preferable toshow a gradual transition between an undistorted state (e.g., state 592)and a distorted state (e.g., state 494), or from a distorted state to anundistorted state, to provide a more natural feel and avoid theappearance of abrupt changes in the display. Intermediate states, suchas states that may occur between state 592 and state 594, can showgradually increasing or decreasing degrees of distortion, asappropriate. As another example, a UI system can provide a boundaryeffect by compressing the elements 542 and 544 shown in display state592 without moving the elements 542 and 544 to the left in the displayarea 300.

Example 6 Changes in Display Orientation

Described techniques and tools can be used on display screens indifferent orientations, such as landscape orientation. Changes indisplay orientation can occur, for example, where a UI has beenconfigured (e.g., by user preference) to be oriented in landscapefashion, or where a user has physically rotated a device. One or moresensors (e.g., an accelerometer) in the device can be used to detectwhen a device has been rotated, and adjust the display orientationaccordingly.

In the example shown in FIG. 5, the display area 300 is oriented inlandscape fashion. Content (e.g., data collection elements 540-544 incontent layer 532) and/or other user interface features in the displayarea 300 can be dynamically adjusted to take into account effects of areorientation (e.g., a new effective width of the display area 300,interpreting directions of user interactions differently, etc.). Forexample, distortion effects can be adjusted, such as by compressing datacollection elements in a horizontal dimension instead of a verticaldimension, to account for display reorientation.

However, such adjustments are not required. For example, if a displayarea has equal height and width, reorientation of the display area to alandscape orientation will not change the effective width of the displayarea.

Example 7 Vertical Boundary Effect with Horizontal Parallax Effect

FIGS. 6A-6E are diagrams showing a content layer 614 that moves intandem with layer 612 above it. In this example, a user 302 (representedby the hand icon) navigates through content layer 614 by interactingwith a touchscreen having the display area 300. The interaction caninclude, for example, contacting the touchscreen with a fingertip,stylus or other object and moving it (e.g., with a flicking or sweepingmotion) across or along the surface of the touchscreen. The contentlayer 614 includes game icons 640, 642, 644, lists 650, 652, 654, andavatar 630 (which is described in more detail below in Example 8). Theother layers 610, 612 include text information (“Games” in layer 610;“Spotlight,” “Xbox Live, “Requests” and “Collection” in layer 612).

The direction of motion that can be caused by user 302 is indicated by aleft-pointing arrow and a right-pointing arrow in FIGS. 6A-6E, alongwith additional up- and down-pointing arrows in FIGS. 6A and 6E. Theright-pointing and left-pointing arrows indicate possible movements(left or right horizontal movements) of the layers 610, 612, 614 inresponse to user movements. In addition to movements of entire layers, auser also can cause movements in elements or parts of layers, dependingon the data in the layer and how the layer is arranged. For example, auser can cause movements (e.g., vertical movements) in layer elements(e.g., lists in a content layer) that are orthogonal to movements (e.g.,horizontal movements) that can be caused in a layer as a whole. Such caninclude scrolling vertically in a list embedded in a content layer thatmoves horizontally. Alternatively, a system that presents layers thatmove vertically can allow horizontal movements in layer elements.

The up-pointing and down-pointing arrows indicate possible movements ofthe list 650 in response to user movements. The amount of movement oflist 650 can be a function of the size or rate of the motion made byuser 302, and the data in list 650. Thus, scrolling of the list 650 canbe element-by-element, page-by-page of elements, or something in betweenthat depends on size or rate of the motion. In this example, list 650includes only one element that is not visible in the display area 300,as shown in FIG. 6A, so a range of small or large downward movements maybe enough to scroll to the end of list 650. In the example shown in FIG.6E, an upward user movement has caused a boundary effect in list 650, inwhich the text of elements in the list are squeezed or compressed in avertical dimension. This effect gives user 302 an indication that theend of the list has been reached.

In this example, the amount of movement in layers 610, 612, 614 is afunction of the data in the layers and the size or rate of the motionmade by the user. Horizontal movement in layers 610, 612, 614 proceedsaccording to the following rules, except during wrapping animations:

-   -   1. The horizontal movement of content layer 614 is locked to        layer 612.    -   2. Layers 612 and 614 will each move at approximately three        times the rate of layer 610, which is approximately ⅓ the length        of layers 612 and 614.

Movement in the layers 610, 612, 614 may differ from the rules describedabove in some circumstances. In the example shown in FIGS. 6A-6E,wrapping is permitted. The arrows indicate that a user can navigate leftfrom the beginning of the content layer 614 (the position shown in FIG.6A and FIG. 6E), and can navigate right from the end of the contentlayer 614 (the position shown in FIG. 6D). During a wrapping animation,some layers may move faster or slower than during other kinds ofmovements. For example, the text in layer 610 can move faster whenwrapping back to the beginning of content layer 614. In FIG. 6D, displayarea 300 shows portions of two letters in layer 610, at the end of the“Games” text string. A wrapping animation to return to the state shownin FIG. 6A can include bringing the data in layers 610, 612, 614(including the text of layer 610) into view from the right, resulting ina more rapid movement in layer 610 than in other contexts, such as atransition from the state shown FIG. 6A to the state shown in FIG. 6B.

In FIGS. 6A-6E, example lock points “A,” “B,” “C” and “D” are indicatedfor layers 610 and 612. In terms of horizontal motion, content layer 614is locked to layer 612; the lock points indicated for layer 612 alsoapply to layer 614. The lock points for each layer indicate thecorresponding position of the left edge of the display area 300 on eachlayer. For example, when a user navigates to a position on content layer614 such that the left edge of the display area 300 is at lock point“A,” the left edge of display area 300 also is aligned at lock point “A”of the other layers 610, 612, as shown in FIGS. 6A and 6E. In FIG. 6B,the left edge of the display area 300 is at lock point “B” in each ofthe layers 610, 612, 614. In FIG. 6C, the left edge of the display area300 is at lock point “C” in each of the layers 610, 612, 614. In FIG.6D, the left edge of the display area 300 is at lock point “D” in eachof the layers 610, 612, 614.

The lock points shown in FIGS. 6A-6E are not generally representative ofa complete set of lock points, and are limited to lock points “A,” “B,”“C” and “D” only for brevity. For example, right-edge lock points can beadded to obtain alignment with the right edge of display area 300, orcenter lock points can be added to obtain alignment with the center ofdisplay area 300. Alternatively, fewer lock points can be used, morelock points can be used, or lock points can be omitted.

User 302 can move left or right in content layer 614 after making an upor down movement in list 650. The current position of list 650 can besaved, or the system can revert to a default position (e.g., thetop-of-list position indicated in FIG. 6A) when navigating left or rightin content layer 614 from list 650. Although the arrows in FIGS. 6A-6E(and other figures) that indicate possible movements are shown forpurposes of explanation, the display area 300 can itself displaygraphical indicators (such as arrows or chevrons) of possible movementsfor the layers and/or list.

The system can interpret user movements to the left or right, evendiagonal movements extending above or below the horizontal plane, as avalid leftward or rightward motion. Similarly, the system can interpretupward or downward movements, even diagonal movement extending to theleft or right of the vertical plane, as a valid upward or downwardmotion. Although FIGS. 6A-6E show the user 302 interacting with aportion of the display area 300 that corresponds to the content layer614, the system also allows interaction with other parts of thetouchscreen (e.g., those that correspond to display area occupied byother layers) to cause movement in the layers 610, 612, 614, list 650,or other UI elements.

Example 8 Avatar

In FIGS. 6A-6E, avatar 630 can provide a visual cue to indicate arelationship between or draw attention to parts of the content layer614.

In FIG. 6B, avatar 630 is positioned between list 652 and list 654. InFIG. 6C, avatar 630 floats behind the text of list 654, but remainscompletely within display area 300. In FIG. 6D, avatar 630 is onlypartially within display area 300; the part that is within the displayarea floats behind game icons 640, 642, 644. The positioning of avatar630 at the left edge of display area 300 can indicate to the user 302that information associated with avatar 630 is available if the user 302navigates in the direction of avatar 630. Avatar 630 can move at varyingspeeds. For example, avatar 630 moves faster in the transition betweenFIGS. 6B and 6C than it does in the transition between FIGS. 6C and 6D.

Alternatively, avatar 630 can move in different ways, or exhibit otherfunctionality. For example, a UI system can present a distortion effectin avatar 630 to indicate a user's location in a data collection withwhich the avatar is associated. Avatar 630 also can be locked toparticular position (e.g., a lock point) in content layer 614 or in someother layer, such that avatar 630 moves at the same horizontal rate asthe layer to which it is locked. As another alternative, avatar 630 canbe associated with a list that can be scrolled up or down, such as list650, and move up or down as the associated list is scrolled up or down.

IV. Detailed Implementation

In this section, a detailed implementation is described comprisingaspects of motion feedback including boundary effects and parallaxeffects, with reference to the following detailed example.

Example 9 Detailed Example

In this detailed example, a set of equations, coefficients and rules aredescribed that can allow a UI system (e.g., a UI system provided as partof a mobile device operating system) to interpret user input such astouch gestures (including multi-touch gestures with more than one touchcontact point) and generate motion feedback in response to user input.Features described in this detailed example include inertia movement,panning and zooming operations, boundary effects, parallax effects, andcombinations thereof. Described features can help to providenatural-looking, smooth motion in response to user input (e.g., touchgestures).

In this detailed example, processing tasks can be handled by differentsoftware modules. For example, a module called “ITouchSession” providescoefficients, gesture positions, and gesture velocity information, and adynamic motion module in a mobile device operating system usesinformation provided by ITouchSession to generate motion feedback (e.g.,parallax effects, boundary effects, etc.). Preferably, gestureinformation provided to the dynamic motion module is accurate (e.g.,with minimal jitter in position information), detailed (e.g., with timestamps on touch input), and low-latency (e.g., under 30 ms). Theinformation (e.g., motion feedback information) generated by the dynamicmotion module can be used by other modules, as well. For example, webbrowsers or other applications that run on the mobile device operatingsystem can use information generated by the dynamic motion module.

In this detailed example, the dynamic motion resulting from userinteraction is defined by a set of motion rules. The motion rules definehow different visual elements react on screen in response to differentgestures. For example, some rules apply to finger-tracking gestures suchas panning or dragging gestures, some rules apply to flick or tossgestures, and some rules apply to pinch or stretch gestures.Additionally, some rules, such as inertia rules, may apply to more thanone type of gesture. The specific motion rules that apply to differentUI elements (or “controls”) are determined by factors such as thecontrol type and control content; not all motion rules will apply to allUI elements. For example, rules for pinch and stretch gestures do notapply to UI elements where pinch and stretch gestures are notrecognized. The motion resulting from the application of motion rules tothe input stream generated by the user can be further refined by anoptional set of modifiers, which are collectively called “optionalmotion features.”

In this detailed example, touch input interactions that result indynamic motion comply with the motion rules. Additionally, different UIelements (or “controls”) can apply zero or more optional motionfeatures, which can be determined by factors such as the desired motion,control type and control content. For example, a list control may opt toenhance motion feedback with boundary effects, while a panorama controlmay apply a parallax feature to some of its layers.

In addition, when a user interacts with a UI element, it can be helpfulto provide some immediate (or substantially immediate) visual feedbackto the user (e.g., a change in movement in the UI element, or some othereffect such as a tilt or highlight). Immediate or substantiallyimmediate feedback helps the user to know that the user interface isresponsive to the user's actions.

In this detailed example, the following motion rules apply in UIelements where the rules (e.g., rules relating to finger-trackinggestures, inertia, boundaries, pinch/stretch gestures) are relevant tothe types of motion that are permitted in the respective UI elements.The motion rules can be modified for some UI elements, such as whereoptional motion features apply to a UI element.

Motion Rule: Finger Tracking

For finger tracking movements (e.g., movements caused by dragging orpanning gestures), the content at the initial gesture point moves indirect correspondence to the gesture. For example, content under theuser's finger at an initial touch point moves with the user's fingerduring the gesture. The current position of a visual element is given bythe following equation:

p=p ₀+(q−q ₀)  (Eq. 1)

where p is the (x, y) vector that represents the current position of thevisual element, p₀ is the (x₀, y₀) vector that represents the visualelement position at the beginning of the gesture, q is the (x, y) vectorthat represents the current touch contact position, and q₀ is the (x₀,y₀) vector that represents the touch contact position at the beginningof the gesture.

Motion Rule: Inertia

In a UI element that allows inertia movement (e.g., a scrolling list),when the user finishes a gesture (e.g., by lifting a finger or otherobject to end the interaction with the touchscreen), a velocity anddirection for that movement is identified, and the motion initiallycontinues in the same direction and speed as the gesture, as if thevisual element was a real, physical object with a non-zero mass. If themotion is not stopped for some other, permissible reason (e.g., wherethe UI element reaches a boundary or is stopped by another usergesture), the motion gradually decelerates over time, eventually comingto a stop. The deceleration proceeds according to a combination ofequations and coefficients, which can vary depending on implementation.Default system-wide coefficient values can be made available. Defaultsystem-wide coefficients can help to maintain a consistent feelingacross all controls. Alternatively, different equations or coefficientscan be used, such as where a particular control has its own frictioncoefficient for modeling different kinds of motion.

The velocity (e.g., in pixels/second) at the end of the gesture iscomputed by the following equation:

v=(q−q ₀)/(t−t ₀),  (Eq. 2)

where v is the (v_(x), v_(y)) velocity vector that represents theinertia velocity at the end of the gesture, q is the (x, y) vector thatrepresents the touch contact position at the end of the gesture, q₀ isthe (x₀, y₀) vector that represents the touch contact position at thetime t₀, t is the timestamp of the last touch input of the gesture, andt₀ is the timestamp of the least recent touch input that happened withinsome fixed period of time from the last touch input. Alternatively, thevelocity can be calculated in another way. For example, a weighted sumof velocities at different time instances can be calculated, withgreater weighting for velocities at the end of the gesture. In thisdetailed example, calculating the velocity is the responsibility ofITouchSession. However, velocity calculations can be handled by othermodules.

The duration of the inertia motion can be computed according to thefollowing equation:

$\begin{matrix}{t_{\max} = \frac{\ln \; \frac{\gamma}{v_{0}}}{\ln \mspace{14mu} \mu}} & \left( {{Eq}.\mspace{14mu} 3} \right)\end{matrix}$

where t_(max) is the duration of the inertia motion, |v₀| is themagnitude of the initial velocity vector (|v₀|>γ), μ is a frictioncoefficient (e.g., MotionParameter_Friction, 0<μ<1), and y is a parkingspeed coefficient (e.g., MotionParameter_ParkingSpeed, 0<γ<|v₀|) that isused to indicate a threshold velocity, below which inertia motion willstop. In one implementation, the friction coefficient is 0.4, and theparking speed coefficient is 60.0. The duration is computed at the startof the inertia motion, and need not be computed again.

The following equation will compute the current velocity vector v at anygiven time t:

v=V ₀·μ^(t)  (Eq. 4).

The new position p′ for the visual element can be computed based on itslast known position p and the time elapsed since the last positionupdate (Δt), as shown in the following equation:

p′=p+v·Δt  (Eq. 5).

The motion stops once the velocity reaches a value smaller than γ.

The actual calculation of values relating to inertia motion (e.g.,velocity, etc.) can differ depending on implementation.

Motion Rule: Interacting with an Element in Inertia Motion

If a new gesture begins while a UI element is in inertia motion, theinertia motion is immediately interrupted. Depending on the new gesture,the motion in the UI element may be stopped, or a new motion may start.If the new gesture causes a new motion in the UI element, the newgesture controls the UI element's motion. The previous gesture and anyconsequent inertia do not affect the motion generated by the newgesture. Handling of new gestures during inertia motion can be differentdepending on implementation. For example, new gestures can be ignoredduring inertia motion or can have different effects on inertia motion.

Motion Rule: Gesture Boundaries

The motion of some UI elements is limited by gesture boundaries. Thedimensions of gesture boundaries and the effects of exceeding gestureboundaries can differ depending on several factors, such as the contentof a UI element and/or a minimum visible area of the UI element. Forexample, lists which don't wrap around indefinitely may only be able toscroll a certain distance based on the number of items in the list and aminimum amount of visible items (e.g., an amount of items that occupiesmost or all of a display area).

In this detailed example, for an element A having a width w_(A), heighth_(A), total area S_(A) and position p_(A) (x_(A), y_(A)), with aminimum visible area S_(Vmin) (width w_(Vmin), height h_(Vmin))currently at position p_(Vmin) (x_(Vmin), y_(Vmin)), a gesture thatbegins at an initial position q (x_(q), y_(q)) has a rectangular gestureboundary area S_(T) (width w_(T), height h_(T)) at positionp_(T)=(x_(T), y_(T)). The minimum visible area indicates a minimumvisible amount of the control (e.g., a minimum number of list items in ascrollable list), but does not require any particular part of thecontrol to be visible. Therefore, the content of the minimum visiblearea for a particular control can vary depending on, for example, thecontrol's current state (e.g., whether the end or beginning of ascrollable list is currently visible).

Conceptually, the position p_(T) of the gesture boundary area can bedefined according to the following equation:

p _(T) =q+(p _(Vmin) +S _(Vmin))−(p _(A) +S _(A))  (Eq. 6).

The x and y coordinates of the position p_(T) of the gesture boundaryarea are defined according to the following equations:

x _(T) =x _(q)+(x _(Vmin) +w _(Vmin))−(x _(A) +w _(A))  (Eq. 7)

y _(T) =y _(q)+(y _(Vmin) +h _(Vmin))−(y _(A) +h _(A))  (Eq. 8).

The dimensions of the gesture boundary area are defined according to thefollowing equation:

S _(T) =S _(A) −S _(Vmin)=(h _(A) −h _(Vmin) ,w _(A) −w _(Vmin))  (Eq.9).

If the new position of the UI element, resulting from user interactionor simulated inertia or some combination, falls outside the area definedby p_(T)+S_(T), a boundary has been exceeded. An appropriate boundaryfeedback modifier can be applied or the new position can be clamped(i.e., kept within the allowed boundaries).

FIG. 7A shows an example boundary diagram for a control having aposition 710 and area 720. The control has a minimum visible area 740(at position 730). For example, the position 730 of the minimum visiblearea can be located at the top left of a display area. Based on aninitial gesture position 750, a gesture boundary at position 770 andhaving area 780 is calculated.

In FIG. 7B, example post-gesture positions 752, 754 are shown.Post-gesture position 752 is outside the gesture boundary area 780, andcauses boundary feedback. Post-gesture position 754 is inside thegesture boundary 780, and does not cause boundary feedback.

FIG. 8A shows an example boundary diagram for a control corresponding tothe scrollable list shown in FIG. 3. In FIG. 8A, the control at position810 has a control area 820 (width W_(A), height h_(A)). In this detailedexample, the coordinates of the control position are considered to be(0, 0). The control has a minimum visible area 840 (at position 830).For example, the position 830 of the minimum visible area 840 can be atthe top left of a display area. Based on an initial gesture position850, a gesture boundary at position 850 (the same position as theinitial gesture position) is calculated. In this detailed example, thegesture boundary 880 has a height of h_(A)−h_(Vmin), and a width of 0.(Due to space limitations, the boundary 880 as shown in FIG. 8A is notto scale.) Therefore, the gesture boundary 880 is actually a verticalline. Although a control having a gesture boundary area with no widthcould cause a boundary feedback effect with any horizontal movement,boundary feedback can be enabled or disabled on an axis basis (e.g.,permitting boundary feedback for vertical movements but not forhorizontal movements). Such a control also can be a candidate for axislocking, to allow only vertical movements and remove any need forboundary feedback for horizontal movements. Axis locking is explained inmore detail below.

In FIG. 8B, example post-gesture positions 852, 854 are shown.Post-gesture position 852 is outside the gesture boundary area 880, andcauses boundary feedback. For example, referring again to FIG. 3, a UIsystem can present a squeeze or compression effect to indicate that thepost-gesture position is outside the gesture boundary area, as shown instate 392. Post-gesture position 854 is inside the gesture boundary area880, and does not cause boundary feedback.

The actual calculation of values relating to motion boundaries and theeffects of motion boundaries can differ depending on implementation. Forexample, if a wrapping feature is available in a UI element, a boundarycan indicate a position at which a boundary effect will be presented(e.g., to indicate that the end of a list has been reached) withoutpreventing further movement beyond the boundary (e.g., wrapping movementfrom the end of the list back to the beginning of the list).

Motion Rule: Pinch/Stretch

Pinch gestures and stretch gestures are gestures that can change thescale (zoom) of the subject area of a control (e.g., a map or image withzoom capability). Pinch gestures and stretch gestures are considered tobe multi-touch gestures because they typically have multiple points ofinteraction. In a typical pinch or stretch gesture scenario, a userplaces two fingers some distance apart from each other on a touchscreen,and either increases (for a stretch gesture) or decreases (for a pinchgesture) the distance between them.

FIG. 9 is a diagram showing example pinch and stretch gestures. On adevice having a display area 300, a user 302 (represented by a handicon) interacts with a control (e.g., a map with zoom features) having acontrol area 910. From display state 990, the user 302 performs a pinchgesture beginning at touch points 950, 960 and ending at touch points952, 962. This results in a zoomed-out version of the map (relative todisplay state 990) in control area 910 in state 992. From display state992, the user 302 performs a stretch gesture beginning at touch points970, 980 and ending at touch points 972, 982. This results in azoomed-in version of the map (relative to display state 992) in controlarea 910 in state 994. Alternatively, a pinch or stretch gesture canbegin or end at other touch points (e.g., with a greater or lesserdistance between beginning and ending touch points) or can use adifferent orientation of touch points (e.g., horizontal or diagonal).

The scale adjustment caused by a pinch or stretch gesture can berepresented as follows. Let qA₀=(x_(A0), y_(A0)) and qB₀=(x_(B0),y_(B0)) be the positions for initial touch points A and B. The distanced₀ between the two points represents a 100% scale factor, and can becalculated according to the following equation:

d ₀ =|qA ₀ −qB ₀|  (Eq. 10).

The distance d₀ includes a horizontal component x_(d0) and a verticalcomponent y_(d0).

Let qA and qB be updated positions for touch points A and B, and letd=(x_(d), y_(d)) be the distance between them, calculated in a similarmanner. The distance d also includes a horizontal component x_(d) and avertical component y_(d). The scale factor s_(zoom) to apply to the UIelement can be calculated according to the following equation:

$\begin{matrix}{s_{zoom} = {\frac{\;}{_{0}} = {\left( {\frac{x_{d}}{x_{d\; 0}},\frac{y_{d}}{y_{d\; 0}}} \right).}}} & \left( {{Eq}.\mspace{14mu} 11} \right)\end{matrix}$

Note that in Equation 11, the scale s is not isometric, i.e., the X andY axes will be scaled differently. For isometric scaling, the followingequation can be used instead:

$\begin{matrix}{s_{zoom} = {\frac{\sqrt{x_{d}^{2} + y_{d}^{2}}}{\sqrt{x_{d\; 0}^{2} + y_{d\; 0}^{2}}}.}} & \left( {{Eq}.\mspace{14mu} 12} \right)\end{matrix}$

In this case, s_(zoom) is a scalar, so the same factor is applied toboth X and Y components.

Alternatively, a scale factor can be calculated in different ways. Forexample, inertia can be applied to a pinch or stretch gesture (such aswhen the gesture ends with a velocity above a threshold), and the scalefactor can be based at least in part on the inertia of the gesture(e.g., increasing the scale of the zoom when a stretch gesture ends witha velocity above a threshold).

To make zooming feel natural, the scale factor can be applied to azooming point (e.g., a center point between touch points qA and qB). Thezooming point c_(z)=(x_(cz), y_(cz)) can be calculated by averaging thetwo touch contact positions, as shown in the following equation:

$\begin{matrix}{c_{z} = {\frac{{qA} + {qB}}{2}.}} & \left( {{Eq}.\mspace{14mu} 13} \right)\end{matrix}$

Alternatively, a zooming point can be calculated in a different way, ora calculation of a zooming point can be omitted.

A pinch/stretch gesture can also produce position changes (panning) inaddition to scale changes. Panning position changes can occursimultaneously with scale changes. The zooming point calculation inEquation 13 is used when simultaneous panning is not allowed. Ifsimultaneous panning is allowed, the zooming point is calculated usingthe initial touch contact positions qA₀ and qB₀ rather than the updatedtouch contact positions qA and qB. If c_(z0) is the initial zoomingpoint and c_(z) is the updated zooming point, the distanced_(pan)=(x_(dpan), y_(dpan)) between the two zooming points represents apanning offset to be applied to the UI element, as shown in thefollowing equation:

d _(pan) =c _(z) −c _(z0)=(x _(cz) −x _(cz0) ,y _(cz) −y _(cz0))  (Eq.14).

Alternatively, a panning offset can be calculated in a different way, ora panning offset can be omitted.

Optional Motion Features: Overview

Optional motion features can be used (e.g., when requested by a control)to refine or add visual feedback to motion generated by gestures.Optional motion features can depend on control type and content. Forexample, some controls (e.g., a scrolling list) may use an optional axislocking feature that is appropriate for the orientation of the control(e.g., allowing only vertical movements in a vertically scrolling list).Optional motion features can be used in combination with each other andwith various motion rules. For example, a vertically scrolling list canuse an axis locking feature and a boundary effect feature, whilefollowing rules for inertia motion and finger tracking motion. DifferentUI elements can use different combinations of rules and optional motionfeatures, even when the different UI elements are visible at the sametime. For example, a movable layer can use parallax effects but omitboundary effects, while a vertically scrolling list in the movable layercan use boundary effects but omit parallax effects. UI elements of thesame basic type can use different sets of optional motion features. Forexample, a first pair of movable layers can use parallax effects andmove at different rates relative to one another, while a third layerparallel to the first pair remains stationary.

When present, optional motion features act like filters, modifying thevalues generated according to other motion rules, such as the motionrules described above.

Optional Motion Features: Axis Locking

For some controls, it may make sense to permit movement only along aparticular axis. For example, it can be useful to restrict movement of amovable, horizontal UI layer (sometimes referred to as a panoramacontrol) to movements along the X axis, or to restrict movement of avertically scrolling list to movements along the Y axis. In such cases,axis locking can be used as an optional motion feature.

In this detailed example, axis locking is applied to a UI element byusing the relevant equations in the motion rules described above, butonly applying an X or Y component (as appropriate) to the motion of theaxis-locked UI element. Changes to the other component are ignored andnot applied to the UI element's motion.

Alternatively, axis locking can be performed in another way. Forexample, in a UI element such as a wheel element that moves about anaxis such as a Z axis, axis locking can be used to permit onlyrotational motion about the axis. As another alternative, axis lockingcan be omitted.

Optional Motion Features: Parallax Effects

Parallax effects can be applied to controls that present multiple layersof content. In a parallax effect, multiple layers are animateddifferently (e.g., moving at different speeds), but the movements of thelayers are based on the same input stream generated by the user.

In a parallax effect, layers that are animated in response to a gesturemove at different speeds relative to one another. The layer that theuser is interacting with directly (e.g., a content layer) is consideredto be the top layer on a Z axis, that is, the layer that is closest tothe user. Other layers are considered to be lower layers on a Z axis,that is, further away from the user. Examples of a parallax effects canbe seen in FIG. 5 and in FIGS. 6A-6D.

In this detailed example, a top layer reacts directly to the gesture,and the other layers move at increasingly lower speeds the further theyare from the top layer along the Z axis. Mathematically speaking, thatcan be accomplished by applying a scaling factor to the delta between aninitial gesture position and an updated gesture position. The updatedgesture position can be obtained directly from user interaction (e.g.,in a finger tracking gesture such as a panning gesture) or from agesture with simulated inertia (e.g., a flick gesture). If k_(P) is theconstant parallax scaling factor to be applied for a particular layer Lat initial position p_(L)=(x_(L), y_(L)), then the parallaxed positionp_(P)=(x_(P), y_(P)) can be computed according to the followingequation:

p _(P) =p _(L) +k _(P)(q−q ₀)  (Eq. 15)

where q is the (x, y) vector that represents the current, post-gestureposition (e.g., after the gesture and application of any simulatedinertia), and q₀ is the (x₀, y₀) vector that represents the touchcontact position at the beginning of the gesture. The parallax constantk_(P) can vary depending on the application, scenario and/or content ofthe control. For example, layers with different lengths can havedifferent parallax constants.

Alternatively, parallax effects can be presented in different ways. Forexample, parallel layers can move according to the model shown inEquation 18 for some movements or parts of a movement and move accordingto other models in other movements or parts of a movement. Referringagain to FIGS. 4A-4C, parallel layers that exhibit parallax effects canmove according to the model shown in Equation 18 in transitions fromFIG. 4A to FIG. 4B, and from FIG. 4B to 4C, and then move according to aspecialized wrapping animation if a gesture to the right from the stateshown in FIG. 4C, or inertia motion from an earlier gesture, causes awrap back to the state shown in FIG. 4A. As another alternative,parallax effects can be omitted.

Optional Motion Features: First Example Boundary Feedback Model

When the boundary feedback motion feature is applied, a boundaryfeedback effect can be applied whenever a gesture would move the UIelement past a boundary, either directly (e.g., by a dragging or panninggesture) or indirectly (e.g., by inertia motion generated by a flickgesture). In this first example boundary feedback model, once the UIelement hits a boundary the content is compressed in the direction ofthe motion (e.g., a vertical compression for a vertical motion) up to acertain threshold. If the compression is caused by inertia, the contentcompresses up to a certain amount based on the velocity at the time theboundary is hit, then decompresses to the original size. If thecompression is caused directly (e.g., by dragging), the compression canbe held as long as the last touch contact point is held and decompresswhen the user breaks contact, or decompress after a fixed length oftime.

In this first example boundary feedback model, the compression effect isachieved by applying a scale factor and dynamically placing acompression point to ensure that the effect looks the same regardless ofthe size of the list. In order to properly compute the motion and scalefor the boundary effect, the first step is to identify that a boundaryhas been crossed and by how much. The boundary motion rule describedabove illustrates how to compute a boundary position in this firstexample boundary feedback model, and in the second example boundaryfeedback model described below.

Let q=(x_(q), y_(q)) be the unmodified, post-gesture position resultingfrom an active finger tracking gesture (e.g., a dragging gesture) orfrom simulated inertia (e.g., from a flick gesture), let x_(L) be theleft boundary, let x_(R) be the right boundary, let y_(T) be the topboundary, and let y_(B) be the bottom boundary. Let r=(r_(x), r_(y))represent how far the post-gesture position exceeds the boundaries withrespect to {x_(L), x_(R), y_(T), y_(B)}:

r _(x)=max(x _(L) −x _(q),0,x _(q) −x _(R))  (Eq. 16)

r _(y)=max(y _(T) −y _(q),0,y _(q) −y _(B))  (Eq. 17)

In cases where only a vertical or horizontal boundary applies (e.g., inaxis-locked elements), r may be calculated in only a vertical orhorizontal dimension, as appropriate, while omitting a calculation ofthe other dimension of r.

Let S_(c) be the compressible area with dimensions (w_(c), h_(c)), whichis some area equal to or greater than the visible area, depending on thevalue of coefficients k % and k₊, where k_(%) is the compressionpercentage coefficient (e.g., MotionParameter_CompressPercent(k_(%)≧0)), and k₊ is the compression offset coefficient (e.g.,MotionParameter_CompressOffset{X,Y} (k₊≧0)). If k_(%)=0, then thecompressible area matches the size of the visible area and the visualresult is that only the visible part of the control is being compressed.If k_(%)=1, the compressible area matches the entire control area.k₊=(w₊, h₊) allows an increase in the compressible area by a fixedamount, regardless of the control area size. In this detailed example,the compressible area can be calculated according to the followingequation:

S _(c) =S _(v) +k _(%)·(S _(A) −S _(V))+k ₊  (Eq. 18),

where S_(A) is the control area with dimensions (W_(A), h_(A)), andS_(V), is the visible area with dimensions (w_(V), h_(V)). In oneimplementation, the compression percentage coefficient is 0.0 and thecompression offset coefficient is 0.5*S_(V).

If the user is actively dragging the content, the compression scalefactor s_(comp)=(s_(compx), s_(compy)) to apply to the target UI elementcan be computed according to the following equations:

$\begin{matrix}{s_{comp} = \frac{S_{c} - {k_{s} \cdot r}}{S_{c}}} & \left( {{Eq}.\mspace{14mu} 19} \right) \\{s_{compx} = \frac{w_{c} - {k_{s} \cdot r_{x}}}{w_{c}}} & \left( {{Eq}.\mspace{14mu} 20} \right) \\{s_{compy} = \frac{h_{c} - {k_{s} \cdot r_{y}}}{h_{c}}} & \left( {{Eq}.\mspace{14mu} 21} \right)\end{matrix}$

where k_(s) is the compression factor coefficient (e.g.,MotionParameter_CompressFactor (0<k_(s)≦1)), and r≦S_(v). In oneimplementation, the compression factor coefficient is 0.2.Alternatively, the scale factor and/or the compressible area can becalculated in different ways. For example, different ranges ofcompression coefficients can be used.

In words, what is being done here is to find the difference between thecompressible area (e.g., in the horizontal or vertical dimensions) andthe amount by which the gesture is compressing the compressible area,then calculating the scale factor based on that difference. Thecompression factor k_(s), if it is less than 1, limits how much thevalue of r (the amount by which the post-gesture position has exceededthe boundary) will cause the compressible area to be compressed. A UIsystem can then place a distortion point (which can also be referred toas a “squeeze point” or “compression point” when applying compressioneffects) at the other side of the compressible area (i.e., the side ofthe compressible area opposite the side where the gesture is being made)and apply that scale factor, resulting in a compression effect.

Once the user ends the dragging gesture (e.g., by lifting a finger fromthe touchscreen), and if no wrap-around functionality is available or ifthe threshold for wrap-around hasn't been reached, the content in thecompressible area returns to a decompressed state. In this first exampleboundary effects model, decompression proceeds according to theappropriate equations set forth below.

In this first example boundary effects model, if a boundary is exceededduring inertia motion, the following equations are used to compute howfar off the boundaries the current position is (r) over time, based onthe velocity at the time the boundary was crossed (v_(h)) and how faroff the boundary the position is (r_(I)) when the following equationsare applied:

$\begin{matrix}{r_{0} = \frac{\min \left( {r_{I},S_{v}} \right)}{k_{s}}} & \left( {{Eq}.\mspace{14mu} 22} \right) \\{a = \frac{v_{h}^{2}}{2 \cdot {\max\left( {{\frac{S_{v}}{k_{s}}r_{I}},0.001} \right)}}} & \left( {{Eq}.\mspace{14mu} 23} \right) \\{r = {r_{0} + {{v_{h}} \cdot t} - {\frac{a \cdot t^{2}}{2}.}}} & \left( {{Eq}.\mspace{14mu} 24} \right)\end{matrix}$

If r<0, the motion is complete. Note that r_(I) can come either frominertia or from an active drag, such as when a user drags the contentinto a compressed state, then flicks, generating inertia.

The compression scale factor s_(inertiacomp)=(s_(inertiacompx),s_(inertiacompy)) to apply during inertia compression can be computedaccording to the following equations:

$\begin{matrix}{s_{inertiacomp} = \frac{s_{c} - r}{s_{c}}} & \left( {{Eq}.\mspace{14mu} 25} \right) \\{s_{inertiacompx} = \frac{w_{c} - r_{x}}{w_{c}}} & \left( {{Eq}.\mspace{14mu} 26} \right) \\{s_{inertiacompy} = \frac{h_{c} - r_{y}}{h_{c}}} & \left( {{Eq}.\mspace{14mu} 27} \right)\end{matrix}$

Note that these equations are similar to the case when dragging thecontent (see Equations 19-21, above), except that the coefficient k_(s)(the compression factor coefficient) has already been applied in thiscase in Equations 22 and 23. Alternatively, the scale factor can becalculated in a different way. For example, constants such as thecompression factor coefficient k_(s) or the value 0.001 in Equation 23can be replaced with other constants depending on implementation.

In this first example boundary effects model, in addition to computingthe scale factor to apply to the target UI element, a compression pointC_(comp)=(c_(compx), c_(compy)) is calculated in order to generate theexpected visual effect. In practice, a compression point can be atdifferent positions in a UI element. For example, a compression pointcan be located at or near the center of a UI element, such that half (orapproximately half) of the content in the UI element will be compressed.As another example, a compression point can be located at or near aborder of UI element, such that all (or approximately all) of thecontent in the UI element will be compressed. The compression point canvary for different UI elements. Using different compression points canbe helpful for providing a consistent amount of distortion in thecontent of UI elements of different sizes. The compression pointposition can be computed according to the following equations:

$\begin{matrix}{c_{compx} = \left\{ \begin{matrix}\left. {left}\Rightarrow{1 - \frac{w_{c}}{w_{A}}} \right. \\\left. {right}\Rightarrow\frac{w_{c}}{w_{A}} \right. \\\left. {none}\Rightarrow 0.5 \right.\end{matrix} \right.} & \left( {{Eq}.\mspace{14mu} 28} \right) \\{c_{compy} = \left\{ \begin{matrix}\left. {top}\Rightarrow{1 - \frac{h_{c}}{h_{A}}} \right. \\\left. {bottom}\Rightarrow\frac{h_{c}}{h_{A}} \right. \\\left. {none}\Rightarrow 0.5 \right.\end{matrix} \right.} & \left( {{Eq}.\mspace{14mu} 29} \right)\end{matrix}$

Alternatively, compression points can be calculated in a different way,or the calculation of compression points can be omitted.

Optional Motion Features: Second Example Boundary Feedback Model

In this second example boundary feedback model, the appearance of theboundary feedback can be controlled in finer detail by using morecoefficients. Also, regardless of whether the compression is causeddirectly (e.g., by dragging) or by inertia, the same calculations areused for the compression effects

Let q=(x_(q), y_(q)) be the unmodified, post-gesture position resultingfrom an active finger tracking gesture (e.g., a dragging gesture) orfrom simulated inertia (e.g., from a flick gesture), let x_(L) be theleft boundary, let x_(R) be the right boundary, let y_(T) be the topboundary, and let y_(B) be the bottom boundary. Let r=(w_(r), h_(r))represent how far the post-gesture position exceeds the boundaries withrespect to {x_(L), x_(R), y_(T), y_(B)}:

w _(r)=max(x _(L) −x _(q),0,x _(q) −x _(R))  (Eq. 30)

h _(r)=max(y _(T) −y _(q),0,y _(q) −y _(B))  (Eq. 31)

In cases where only a vertical or horizontal boundary applies (e.g., inaxis-locked elements), r may be calculated in only a vertical orhorizontal dimension, as appropriate, while omitting a calculation ofthe other dimension of r.

As in the first example boundary effects model, S_(c) is thecompressible area with dimensions (w_(c), h_(c)), calculated as shown inEquation 18. However, in this second example boundary effects model,given r=(w_(r), h_(r)) and a compressible area S_(c)=(w_(c), h_(c)), thecompression scale factor s_(comp)=(s_(compx), s_(compy)) to apply to thetarget UI element is computed according to the following equations:

$\begin{matrix}{d = \left( {{\min \left( {w_{r},w_{c}} \right)},{\min \left( {h_{r},h_{c}} \right)}} \right)} & \left( {{Eq}.\mspace{14mu} 32} \right) \\{F_{s} = {k_{s} \cdot d^{k_{e}}}} & \left( {{Eq}.\mspace{14mu} 33} \right) \\{r^{\prime} = {d - {{F_{s} \cdot \left( {1 - k_{d}} \right) \cdot \Delta}\; t}}} & \left( {{Eq}.\mspace{14mu} 34} \right) \\{r^{\prime\prime} = {\max \left( {0,{\min \left( {k_{L},r^{\prime}} \right)}} \right)}} & \left( {{Eq}.\mspace{14mu} 35} \right) \\{s_{comp} = \frac{s_{c} - r^{\prime\prime}}{s_{c}}} & \left( {{Eq}.\mspace{14mu} 36} \right) \\{s_{compx} = \frac{w_{c} - w_{r}^{\prime\prime}}{w_{c}}} & \left( {{Eq}.\mspace{14mu} 37} \right) \\{s_{compy} = \frac{h_{c} - h_{r}^{\prime\prime}}{h_{c}}} & \left( {{Eq}.\mspace{14mu} 38} \right)\end{matrix}$

where k_(s) is a spring factor coefficient (e.g.,MotionParameter_SpringFactor (k_(s)>0)), k_(e) is a spring powercoefficient (e.g., MotionParameter_SpringPower (k_(e)>0)), k_(d) is adamper factor coefficient (e.g., MotionParameter_DamperFactor(0≦k_(d)≦1)), k_(L) is a compression limit coefficient (e.g.,MotionParameter_CompressionLimit (k_(L)>0)), and Δt is the time intervalsince the last iteration of the simulation (Δt≧0). The equation for r″imposes limits on the movement in the UI element during boundaryfeedback. If r″=0, the motion is considered to be complete.

In this second example boundary effects model, the spring factorcoefficient k_(s) is a number that specifies how much resistance willcounteract the inertia force, and the spring power coefficient k_(e)shapes the curve of the resistance. For example, a spring powercoefficient of 1 indicates linear resistance, where resistance increasesat a constant rate as compression increases. A spring power coefficientgreater than 1 means that the resistance will increase at an increasingrate at higher compression, and less than 1 means that the resistancewill increase, but at a decreasing rate, at higher compression. Thedamper factor coefficient k_(d) represents a percentage of energyabsorbed by the system and taken away from the inertia. The damperfactor coefficient can be used to smooth out the boundary effect andavoid a repeated cycle of compression and decompression. The timeinterval Δt can vary depending on the number of frames per second in theanimation of the boundary feedback, hardware speed, and other factors.In one implementation, the time interval is about 16 ms between eachupdate. Varying the time interval can alter the effect of the boundaryeffect. For example, a smaller time interval can result in more fluidmotion.

Alternatively, the scale factor and/or the compressible area can becalculated in different ways. For example, different ranges or values ofcoefficients can be used.

FIG. 10 is a graph of position changes in a UI element over timeaccording to the second example boundary effects model. According to thegraph shown in FIG. 10, a compression effect occurs during the time thatthe position of the UI element exceeds the boundary position indicatedby the dashed line 1010 in FIG. 10). The compression line can indicatethe position of a boundary in a UI element.

The shape of the position curve 1020 can be modified in different ways,such as by adjusting coefficients. For example, by adjusting the springpower coefficient, the uppermost tip of the boundary effect curve 1020can be made to go higher (e.g., up to a configurable limit) or lower fora particular initial velocity. A higher tip of the curve can indicate agreater compression effect, and a lower tip can indicate a lessercompression effect. As another example, by adjusting the spring factorcoefficient, the duration of the compression can be adjusted to beshorter or longer. In FIG. 10, the duration is represented by thedistance between the points at which the line 1010 is crossed by thecurve 1020. As another example, by adjusting the damper factorcoefficient the right-hand tail of the curve (e.g., the part of thecurve 1020 after the boundary position line 1010 is crossed for thesecond time) can be moved up or down, resulting in a more gradual ormore abrupt end to the compression effect. Coefficients can be adjustedin combination or independently, and other values besides thoseindicated can be adjusted as well, to cause changes in position.Different combinations of adjustments can be used to obtain specificshapes in the position curve 1020.

In this second example boundary effects model, a current inertiavelocity v_(c) and a current touch contact position q_(c) can be updatedto reflect the physics interaction of the boundary effect. For example,the updated velocity v′_(c) and updated touch contact position q′_(c)are calculated according to the following equations:

$\begin{matrix}{v_{n} = {\max\left( {0,{v_{c} - \left( {{F_{s} \cdot k_{d}} + {\max\left( {0,\frac{r^{\prime} - k_{L}}{\Delta \; t}} \right)}} \right)}} \right)}} & \left( {{Eq}.\mspace{14mu} 39} \right) \\{v_{c}^{\prime} = \left\{ \begin{matrix}{v_{n},} & {r^{\prime\prime} > 0} \\{0,} & {r^{\prime\prime} \leq 0}\end{matrix} \right.} & \left( {{Eq}.\mspace{14mu} 40} \right) \\{q_{c}^{\prime} = \left\{ \begin{matrix}{{q_{c} - r^{\prime}},} & {r^{\prime\prime} \geq 0} \\{{q_{c} - d},} & {r^{\prime\prime} < 0}\end{matrix} \right.} & \left( {{Eq}.\mspace{14mu} 41} \right)\end{matrix}$

Various alternatives to the boundary feedback models described above arepossible. For example, if wrapping beyond a boundary (e.g., wrappingback to the beginning of a list after the end of the list has beenreached) is permitted, if the compression is caused by dragging, thelist can wrap around once a threshold compression has been reached. Asanother alternative, boundary effects can be omitted.

Reference Values:

A UI system can provide programmatic access to system-wide values e.g.,(inertia values, boundary effect values). Using system-wide values canhelp in maintaining consistent UI behavior across components andframeworks, and can allow adjustments to the behavior in multiple UIelements at once. For example, inertia effects in multiple UI elementscan be changed by adjusting system-wide inertia values.

In one implementation, in order to provide frameworks with access to thereference values of each coefficient, an API is included theITouchSession module (HRESULT GetMotionParameterValue(IN MotionParameterID, OUT float*value)). In one implementation, the identifiers anddefault values for the coefficients whose values are accessible throughthe ITouchSession::GetMotionParameterValue( ) API are as follows:

enum MotionParameter {  MotionParameter_Friction, // default: 0.4f MotionParameter_ParkingSpeed, // default: 60.0f MotionParameter_MaximumSpeed, // default: 20000.0f MotionParameter_SpringFactor, // default: 48.0f MotionParameter_SpringPower, // default: 0.75f MotionParameter_DamperFactor, // default: 0.09f MotionParameter_CompressLimit, // default: 300.0f MotionParameter_CompressPercent, // default: 0.0f MotionParameter_CompressOffsetX, // default: 720.0f MotionParameter_CompressOffsetY, // default: 1200.0f };The values that are accessible through the API can vary depending onimplementation. For example, a UI system that uses the first exampleboundary effects model described above can omit values such as springfactor, spring power, and damper factor values. Or, a UI system can useadditional values or replace the listed default values with otherdefault values. Values can be fixed or adjustable, and can be updatedduring operation of the system (e.g., based on system settings or userpreferences).

Example 10 UI System

FIG. 11 is a system diagram showing an example UI system 1100 thatpresents a UI on a device (e.g., a smartphone or other mobile computingdevice). In this example, the UI system 1100 is a multi-layer UI systemthat presents motion feedback (e.g., parallax effects, boundary effects,etc.). Alternatively, the system 1100 presents motion feedback in UIsthat do not have multiple UI layers. The system 1100 can be used toimplement functionality described in other examples, or otherfunctionality.

In this example, the system 1100 includes a hub module 1110 thatprovides a declarative description of a hub page to UI control 1120,which controls display of UI layers. UI control 1120 also can bereferred to as a “panorama” or “pano” control in a multi-layer UIsystem. Such a description can be used, for example, when the UI layersmove in a panoramic, or horizontal, fashion. Alternatively, UI control1120 controls UI layers that move vertically, or in some other fashion.UI control 1120 includes markup generator 1130 and motion module 1140.

The declarative description of the hub page includes information thatdefines UI elements. In a multi-layer UI system, UI elements can includemultiple layers, such as a background layer, a title layer, a sectionheader layer, and a content layer. The declarative description of thehub page is provided to markup generator 1130, along with otherinformation such as style information and/or configuration properties.Markup generator 1130 generates markup that can be used to render the UIlayers. Motion module 1140 accepts events (e.g., direct UI manipulationevents) generated in response to user input and generates motioncommands. The motion commands are provided along with the markup to a UIframework 1150. In the UI framework 1150, the markup and motion commandsare received in layout module 1152, which generates UI renderingrequests to be sent to device operating system (OS) 1160. The device OS1160 receives the rendering requests and causes a rendered UI to beoutput to a display on the device. System components such as hub module1110, UI control 1120, and UI framework 1150 also can be implemented aspart of device OS 1160. In one implementation, the device OS 1160 is amobile computing device OS.

A user (not shown) can generate user input that affects how the UI ispresented. In the example shown in FIG. 11, the UI control 1120 listensfor direct UI manipulation events generated by UI framework 1150. In UIframework 1150, direct UI manipulation events are generated byinteraction module 1154, which receives gesture messages (e.g., messagesgenerated in response to panning or flick gestures by a user interactingwith a touchscreen on the device) from device OS 1160. Interactionmodule 1154 also can accept and generate direct UI manipulation eventsfor navigation messages generated in response to other kinds of userinput, such as voice commands, directional buttons on a keypad orkeyboard, trackball motions, etc. Device OS 1160 includes functionalityfor recognizing user gestures and creating messages that can be used byUI framework 1150. UI framework 1150 translates gesture messages intodirection UI manipulation events to be sent to UI control 1120.

The system 1100 can distinguish between different gestures on thetouchscreen, such as drag gestures, pan gestures and flick gestures. Thesystem 1100 can also detect a tap or touch gesture, such as where theuser touches the touchscreen in a particular location, but does not movethe finger, stylus, etc. before breaking contact with the touchscreen.As an alternative, some movement is permitted, within a small threshold,before breaking contact with the touchscreen in a tap or touch gesture.

The system 1100 interprets an interaction as a particular gesturedepending on the nature of the interaction with the touchscreen. Thesystem 1100 obtains one or more discrete inputs from a user'sinteraction. A gesture can be determined from a series of inputs. Forexample, when the user touches the touchscreen and begins a movement inUI element in a horizontal direction while maintaining contact with thetouchscreen, the system 1100 can fire a pan input and begin a horizontalmovement in the UI element. The system 1100 can continue to tire paninputs while the user maintains contact with the touchscreen andcontinues moving. For example, the system 1100 can fire a new pan inputeach time the user moves N pixels while maintaining contact with thetouch screen. In this way, a continuous physical gesture on atouchscreen can be interpreted by the system 1100 as a series of paninputs. The system 1100 can continuously update the contact position andrate of movement. When the physical gesture ends (e.g., when user breakscontact with the touchscreen), the system 1100 can determine whether tointerpret the motion at the end as a flick by determining how quicklythe user's finger, stylus, etc., was moving when it broke contact withthe touchscreen, and whether the rate of movement exceeds a threshold.

The system 1100 can render motion (e.g., motion in a layer, list, orother UI element) on the display differently depending on the type ofgesture. For example, in the case of a horizontal drag gesture (in whichthe user is currently maintaining contact with the touchscreen) on acontent layer in a multi-layer UI system, the system 1100 moves thecontent layer in a horizontal direction by the same distance as thehorizontal distance of the drag. In a parallax effect, the title layerand background layer also move in response to the drag. As anotherexample, in the case of a pan gesture (in which the user has ended thegesture) on the content layer, the system 1100 can move the contentlayer in the amount of the pan, and determine whether to perform anadditional movement in the content layer. For example, the system 1100can perform a locking animation (i.e., an animation of a movement in thecontent layer to snap to a lock point) and move the content layer to aleft or right lock point associated with an item in the content layer.The system 1100 can determine which lock point associated with thecurrent pane is closer, and transition to the closer lock point. Asanother example, the system 1100 can move the content layer in order tobring an item in the content layer that is in partial view on thedisplay area into full view. Alternatively, the system 1100 can maintainthe current position of the content layer. As another example, in thecase of a flick gesture (e.g., where the user was moving more rapidlywhen the user broke contact with the touchscreen) on the content layer,the system 1100 can use simulated inertia to determine a post-gestureposition for the content layer. Alternatively, the system 1100 canpresent some other kind of motion, such as a wrapping animation or othertransition animation. The threshold velocity for a flick to be detected(i.e., to distinguish a flick gesture from a pan gesture) can varydepending on implementation.

The system 1100 also can implement edge tap functionality. In an edgetap, a user can tap within a given margin of edges of the display areato cause a transition (e.g., to a next or previous item in a contentlayer, a next or previous list element, etc.). This can be useful, forexample, where an element is partially in view in the display area. Theuser can tap near the element to cause the system to bring that elementcompletely into the display area.

V. Extensions and Alternative Implementations

Various extensions and alternatives to the embodiments described hereinare possible.

For example, described examples show different positions of UI elements(e.g., layers, lists, etc.) that may be of interest to a user. A usercan begin navigation of an element at the beginning of an element, oruse different entry points. For example, a user can begin interacting inthe middle of a content layer, at the end of a content layer, etc. Thiscan be useful, for example, where a user has previously exited at aposition other than the beginning of a layer (e.g., the end of a layer),so that the user can return to the prior location (e.g., before andafter a user uses an application (such as an audio player) invoked byactuating a content image).

As another example, other models can be used to model inertia andmovement. For example, although some equations are provided in someexamples that approximate motion according to Newtonian physics, otherequations can be used that model other kinds of motion (e.g.,non-Newtonian physics).

As another example, although controls can share global parameters, suchas a global friction coefficient for inertia motion, parameters can becustomized. For example, friction coefficients can be customized forspecific controls or content, such as friction coefficients that resultin more rapid deceleration of inertia motion for photos or photo slideshows.

As another example, boundary feedback can be applied to pinch andstretch gestures. Such boundary feedback can useful, for example, toindicate that a border of the UI element has been reached.

As another example, additional feedback on gestures can be used. Forexample, visual feedback such as a distortion effect can be used toalert a user that a UI element with zoom capability (e.g., a map orimage) has reached a maximum or minimum zoom level.

As another example, boundary effects such as compression effects canthemselves produce inertia movement. For example, when a verticallyscrolling list is compressed upon reaching the end of the list, andbreaking contact with the touchscreen causes the list decompress, thedecompression can be combined with a spring or rebound effect, causingthe list to scroll in the opposite direction of the motion thatoriginally caused the compression. In this way, the spring effect couldprovide boundary feedback to indicate that the end of list had beenreached, while also providing an alternative technique for navigatingthe list. The spring effect could be used to cause a movement in thelist similar to a flick in the opposite direction. Inertia motion canapplied to motion caused by the spring effect.

VI. Example Computing Environment

FIG. 12 illustrates a generalized example of a suitable computingenvironment 1200 in which several of the described embodiments may beimplemented. The computing environment 1200 is not intended to suggestany limitation as to scope of use or functionality, as the techniquesand tools described herein may be implemented in diverse general-purposeor special-purpose computing environments.

With reference to FIG. 12, the computing environment 1200 includes atleast one CPU 1210 and associated memory 1220. In FIG. 12, this mostbasic configuration 1230 is included within a dashed line. Theprocessing unit 1210 executes computer-executable instructions and maybe a real or a virtual processor. In a multi-processing system, multipleprocessing units execute computer-executable instructions to increaseprocessing power. FIG. 12 shows a second processing unit 1215 (e.g., aGPU or other co-processing unit) and associated memory 1225, which canbe used for video acceleration or other processing. The memory 1220,1225 may be volatile memory (e.g., registers, cache, RAM), non-volatilememory (e.g., ROM, EEPROM, flash memory, etc.), or some combination ofthe two. The memory 1220, 1225 stores software 1280 for implementing asystem with one or more of the described techniques and tools.

A computing environment may have additional features. For example, thecomputing environment 1200 includes storage 1240, one or more inputdevices 1250, one or more output devices 1260, and one or morecommunication connections 1270. An interconnection mechanism (not shown)such as a bus, controller, or network interconnects the components ofthe computing environment 1200. Typically, operating system software(not shown) provides an operating environment for other softwareexecuting in the computing environment 1200, and coordinates activitiesof the components of the computing environment 1200.

The storage 1240 may be removable or non-removable, and includesmagnetic disks, magnetic tapes or cassettes, CD-ROMs, DVDs, memorycards, or any other medium which can be used to store information andwhich can be accessed within the computing environment 1200. The storage1240 stores instructions for the software 1280 implementing describedtechniques and tools.

The input device(s) 1250 may be a touch input device such as a keyboard,mouse, pen, trackball or touchscreen, an audio input device such as amicrophone, a scanning device, a digital camera, or another device thatprovides input to the computing environment 1200. For video, the inputdevice(s) 1250 may be a video card, TV tuner card, or similar devicethat accepts video input in analog or digital form, or a CD-ROM or CD-RWthat reads video samples into the computing environment 1200. The outputdevice(s) 1260 may be a display, printer, speaker, CD-writer, or anotherdevice that provides output from the computing environment 1200.

The communication connection(s) 1270 enable communication over acommunication medium to another computing entity. The communicationmedium conveys information such as computer-executable instructions,audio or video input or output, or other data in a modulated datasignal. A modulated data signal is a signal that has one or more of itscharacteristics set or changed in such a manner as to encode informationin the signal. By way of example, and not limitation, communicationmedia include wired or wireless techniques implemented with anelectrical, optical, RF, infrared, acoustic, or other carrier.

The techniques and tools can be described in the general context ofcomputer-readable media. Computer-readable media are any available mediathat can be accessed within a computing environment. By way of example,and not limitation, with the computing environment 1200,computer-readable media include memory 1220, 1225, storage 1240, andcombinations thereof.

The techniques and tools can be described in the general context ofcomputer-executable instructions, such as those included in programmodules, being executed in a computing environment on a target real orvirtual processor. Generally, program modules include routines,programs, libraries, objects, classes, components, data structures, etc.that perform particular tasks or implement particular abstract datatypes. The functionality of the program modules may be combined or splitbetween program modules as desired in various embodiments.Computer-executable instructions for program modules may be executedwithin a local or distributed computing environment. Any of the methodsdescribed herein can be implemented by computer-executable instructionsencoded on one or more computer-readable media (e.g., computer-readablestorage media or other tangible media).

For the sake of presentation, the detailed description uses terms like“interpret” and “squeeze” to describe computer operations in a computingenvironment. These terms are high-level abstractions for operationsperformed by a computer, and should not be confused with acts performedby a human being. The actual computer operations corresponding to theseterms vary depending on implementation.

VII. Example Implementation Environment

FIG. 13 illustrates a generalized example of a suitable implementationenvironment 1300 in which described embodiments, techniques, andtechnologies may be implemented.

In example environment 1300, various types of services (e.g., computingservices 1312) are provided by a cloud 1310. For example, the cloud 1310can comprise a collection of computing devices, which may be locatedcentrally or distributed, that provide cloud-based services to varioustypes of users and devices connected via a network such as the Internet.The cloud computing environment 1300 can be used in different ways toaccomplish computing tasks. For example, with reference to describedtechniques and tools, some tasks, such as processing user input andpresenting a user interface, can be performed on a local computingdevice, while other tasks, such as storage of data to be used insubsequent processing, can be performed elsewhere in the cloud.

In example environment 1300, the cloud 1310 provides services forconnected devices with a variety of screen capabilities 1320A-N.Connected device 1320A represents a device with a mid-sized screen. Forexample, connected device 1320A could be a personal computer such asdesktop computer, laptop, notebook, netbook, or the like. Connecteddevice 1320B represents a device with a small-sized screen. For example,connected device 1320B could be a mobile phone, smart phone, personaldigital assistant, tablet computer, and the like. Connected device 1320Nrepresents a device with a large screen. For example, connected device1320N could be a television (e.g., a smart television) or another deviceconnected to a television or projector screen (e.g., a set-top box orgaming console).

A variety of services can be provided by the cloud 1310 through one ormore service providers (not shown). For example, the cloud 1310 canprovide services related to mobile computing to one or more of thevarious connected devices 1320A-N. Cloud services can be customized tothe screen size, display capability, or other functionality of theparticular connected device (e.g., connected devices 1320A-N). Forexample, cloud services can be customized for mobile devices by takinginto account the screen size, input devices, and communication bandwidthlimitations typically associated with mobile devices.

VIII. Example Mobile Device

FIG. 14 is a system diagram depicting an exemplary mobile device 1400including a variety of optional hardware and software components, showngenerally at 1402. Any components 1402 in the mobile device cancommunicate with any other component, although not all connections areshown, for ease of illustration. The mobile device can be any of avariety of computing devices (e.g., cell phone, smartphone, handheldcomputer, personal digital assistant (PDA), etc.) and can allow wirelesstwo-way communications with one or more mobile communications networks1404, such as a cellular or satellite network.

The illustrated mobile device can include a controller or processor 1410(e.g., signal processor, microprocessor, ASIC, or other control andprocessing logic circuitry) for performing such tasks as signal coding,data processing, input/output processing, power control, and/or otherfunctions. An operating system 1412 can control the allocation and usageof the components 1402 and support for one or more application programs1414. The application programs can include common mobile computingapplications (e.g., email applications, calendars, contact managers, webbrowsers, messaging applications), or any other computing application.

The illustrated mobile device can include memory 1420. Memory 1420 caninclude non-removable memory 1422 and/or removable memory 1424. Thenon-removable memory 1422 can include RAM, ROM, flash memory, a diskdrive, or other well-known memory storage technologies. The removablememory 1424 can include flash memory or a Subscriber Identity Module(SIM) card, which is well known in GSM communication systems, or otherwell-known memory storage technologies, such as smart cards. The memory1420 can be used for storing data and/or code for running the operatingsystem 1412 and the applications 1414. Example data can include webpages, text, images, sound files, video data, or other data sets to besent to and/or received from one or more network servers or other mobiledevices via one or more wired or wireless networks. The memory 1420 canbe used to store a subscriber identifier, such as an InternationalMobile Subscriber Identity (IMSI), and an equipment identifier, such asan International Mobile Equipment Identifier (IMEI). Such identifierscan be transmitted to a network server to identify users and equipment.

The mobile device can support one or more input devices 1430, such as atouchscreen 1432, microphone 1434, camera 1436, physical keyboard 1438and/or trackball 1440 and one or more output devices 1450, such as aspeaker 1452 and a display 1454. Other possible output devices (notshown) can include a piezoelectric or other haptic output device. Somedevices can serve more than one input/output function. For example,touchscreen 1432 and display 1454 can be combined in a singleinput/output device.

Touchscreen 1432 can accept input in different ways. For example,capacitive touchscreens detect touch input when an object (e.g., afingertip or stylus) distorts or interrupts an electrical currentrunning across the surface. As another example, touchscreens can useoptical sensors to detect touch input when beams from the opticalsensors are interrupted. Physical contact with the surface of the screenis not necessary for input to be detected by some touchscreens.

A wireless modem 1460 can be coupled to an antenna (not shown) and cansupport two-way communications between the processor 1410 and externaldevices, as is well understood in the art. The modem 1460 is showngenerically and can include a cellular modem for communicating with themobile communication network 1404 and/or other radio-based modems (e.g.,Bluetooth or Wi-Fi). The wireless modem 1460 is typically configured forcommunication with one or more cellular networks, such as a GSM networkfor data and voice communications within a single cellular network,between cellular networks, or between the mobile device and a publicswitched telephone network (PSSTN).

The mobile device can further include at least one input/output port1480, a power supply 1482, a satellite navigation system receiver 1484,such as a Global Positioning System (GPS) receiver, an accelerometer1486, a transceiver 1488 (for wirelessly transmitting analog or digitalsignals) and/or a physical connector 1490, which can be a USB port, IEEE1494 (firewall) port, and/or RS-232 port. The illustrated components1402 are not required or all-inclusive, as components can be deleted andother components can be added.

In view of the many possible embodiments to which the principles of thedisclosed invention may be applied, it should be recognized that theillustrated embodiments are only preferred examples of the invention andshould not be taken as limiting the scope of the invention. Rather, thescope of the invention is defined by the following claims. We thereforeclaim as our invention all that comes within the scope and spirit ofthese claims.

1. In a computer system, a method comprising: receiving gestureinformation corresponding to a gesture on a touch input device;calculating simulated inertia motion for a movable user interfaceelement based at least in part on the gesture information; based atleast in part on the gesture information and on the simulated inertiamotion, calculating a post-gesture position of the movable userinterface element; determining that the post-gesture position exceeds agesture boundary of the movable user interface element; and calculatinga distortion effect in the movable user interface element to indicatethat the gesture boundary has been exceeded.
 2. The method of claim 1wherein calculating the distortion effect comprises: determining anextent by which the gesture boundary has been exceeded; determining acompressible area of the movable user interface element; and determininga scale factor for the distortion effect based at least in part on thecompressible area and the extent by which the gesture boundary has beenexceeded.
 3. The method of claim 2 further comprising scaling thecompressible area according to the scale factor.
 4. The method of claim2 wherein calculating the distortion effect further comprisesdetermining a distortion point for the distortion effect.
 5. The methodof claim 4 further comprising scaling the compressible area according tothe scale factor and the distortion point.
 6. The method of claim 1wherein the distortion effect is a squeeze effect.
 7. The method ofclaim 1 further comprising: displaying a portion of the movable userinterface element to indicate availability of a wrapping feature in themovable user interface element.
 8. The method of claim 1, wherein thegesture information comprises gesture coordinates.
 9. The method ofclaim 1 wherein calculating the post-gesture position comprisesinterrupting the simulated inertia motion when new gesture informationcorresponding to a new gesture is received.
 10. The method of claim 1wherein calculating simulated inertia motion is further based on inertiainformation comprising an inertia velocity.
 11. The method of claim 10,wherein the inertia velocity is based at least in part on a frictioncoefficient.
 12. The method of claim 10 wherein calculating simulatedinertia motion comprises: comparing the inertia velocity with a parkingspeed coefficient; and determining whether to stop the inertia motionbased on the comparing.
 13. The method of claim 1 wherein the movableuser interface element is an axis-locked user interface element.
 14. Ina computer system, a method comprising: receiving user input thatindicates movement in a graphical user interface element having pluralmovable layers; based at least in part on inertia information and theuser input, calculating a first motion having a first movement rate in afirst layer of the plural movable layers; and calculating a parallaxmotion in a second layer of the plural movable layers, wherein theparallax motion is based at least in part on the first motion, andwherein the parallax motion comprises a movement of the second layer ata second movement rate that differs from the first movement rate. 15.The method of claim 14 wherein calculating the parallax motion is basedat least in part on a parallax constant for the second layer.
 16. Themethod of claim 14 wherein calculating the parallax motion is based atleast in part on an amount of displayable data in the second layer. 17.The method of claim 14 wherein calculating the first motion comprisesapplying simulated inertia motion based at least in part on the inertiainformation.
 18. The method of claim 14 wherein the user input is agesture on a touch screen.
 19. The method of claim 18 wherein theinertia information comprises a velocity of the gesture.
 20. A computerreadable medium having stored thereon computer-executable instructionsoperable to cause a computer to perform a method comprising: receivinggesture information corresponding to a gesture on a touch input device,the gesture information indicating a movement of a user interfaceelement having a movement boundary; based at least in part on thegesture information, computing a new position of the user interfaceelement; based at least in part on the new position, determining thatthe movement boundary has been exceeded; determining an extent by whichthe movement boundary has been exceeded; determining a compressible areaof the user interface element; determining a scale factor for adistortion effect based at least in part on the compressible area andthe extent by which the movement boundary has been exceeded; andpresenting a distortion effect in the user interface element, whereinthe distortion effect comprises a visual compression of content in thecompressible area according to the scale factor, wherein the visualcompression is in a dimension that corresponds to the movement of theuser interface element.